2024-03-19T01:09:14.895 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-19T01:09:14.895 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} --archive /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297 --name pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/orig.config.yaml 2024-03-19T01:09:14.939 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T01:09:15.025 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297 branch: wip-batrick-testing-20240318.181317 description: fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} email: pdonnell@redhat.com first_in_suite: false job_id: '7609297' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-batrick-testing-20240318.181317 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240318.181317 sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 3005 sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 sleep_before_teardown: 0 subset: 433/512 suite: fs suite_branch: wip-batrick-testing-20240318.181317 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEk0CATx6EFmnq8G/eetGLUfRIFzePjKl1vLx1XTY7jtRWcro9dJkpYQ7AbkSZ/cqh/1TFSTXT4N26YAozbh5S4= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBAQ+hAwzR5bVVCCOh/YFbz5HEM/C7nGcxQonnlhWFYGHrOLA8v6OS5DUb26YGAKPfdWzQEgzrs0ki802j8QUmc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_cephfs_shell teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_21:05:49 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-19T01:09:15.025 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa; will attempt to use it 2024-03-19T01:09:15.026 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks 2024-03-19T01:09:15.026 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-19T01:09:15.028 INFO:teuthology.task.internal:Checking packages... 2024-03-19T01:09:15.053 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b6ae24918b03b1b8a19c9263a50655bba80bb810' 2024-03-19T01:09:15.054 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T01:09:15.054 INFO:teuthology.packaging:ref: None 2024-03-19T01:09:15.054 INFO:teuthology.packaging:tag: None 2024-03-19T01:09:15.054 INFO:teuthology.packaging:branch: wip-batrick-testing-20240318.181317 2024-03-19T01:09:15.054 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:09:15.054 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-batrick-testing-20240318.181317 2024-03-19T01:09:15.197 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:09:15.199 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-19T01:09:15.206 INFO:teuthology.task.internal:no buildpackages task found 2024-03-19T01:09:15.206 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-19T01:09:15.212 INFO:teuthology.task.internal:Saving configuration 2024-03-19T01:09:15.228 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-19T01:09:15.234 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T01:09:15.260 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:03:39.824271', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB8pY0ZpQIG5e72alWO84/q0rArmcsfTVRRZDFYHrno+'} 2024-03-19T01:09:15.285 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:03:39.822541', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMVq7/fpxqa4lPDkpQYFR05ONKm61xxEUjrdsbwsJrxQ'} 2024-03-19T01:09:15.285 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-19T01:09:15.291 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-19T01:09:15.291 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-19T01:09:15.291 INFO:teuthology.run_tasks:Running task console_log... 2024-03-19T01:09:15.363 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f84bbb913a0>, signals=[15]) 2024-03-19T01:09:15.363 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-19T01:09:15.369 INFO:teuthology.task.internal:Opening connections... 2024-03-19T01:09:15.369 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:09:15.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:09:15.452 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:09:15.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:09:15.531 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-19T01:09:15.537 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-03-19T01:09:15.542 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-03-19T01:09:15.542 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-03-19T01:09:15.593 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-19T01:09:15.593 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Ubuntu" 2024-03-19T01:09:15.593 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="22.04" 2024-03-19T01:09:15.593 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-19T01:09:15.593 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_CODENAME=jammy 2024-03-19T01:09:15.594 INFO:teuthology.orchestra.run.smithi007.stdout:ID=ubuntu 2024-03-19T01:09:15.594 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE=debian 2024-03-19T01:09:15.594 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-19T01:09:15.594 INFO:teuthology.orchestra.run.smithi007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-19T01:09:15.594 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-19T01:09:15.594 INFO:teuthology.orchestra.run.smithi007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-19T01:09:15.594 INFO:teuthology.orchestra.run.smithi007.stdout:UBUNTU_CODENAME=jammy 2024-03-19T01:09:15.595 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-03-19T01:09:15.615 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2024-03-19T01:09:15.620 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2024-03-19T01:09:15.620 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2024-03-19T01:09:15.672 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-19T01:09:15.672 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="Ubuntu" 2024-03-19T01:09:15.672 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="22.04" 2024-03-19T01:09:15.672 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-19T01:09:15.672 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_CODENAME=jammy 2024-03-19T01:09:15.672 INFO:teuthology.orchestra.run.smithi167.stdout:ID=ubuntu 2024-03-19T01:09:15.672 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE=debian 2024-03-19T01:09:15.672 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-19T01:09:15.672 INFO:teuthology.orchestra.run.smithi167.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-19T01:09:15.673 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-19T01:09:15.673 INFO:teuthology.orchestra.run.smithi167.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-19T01:09:15.673 INFO:teuthology.orchestra.run.smithi167.stdout:UBUNTU_CODENAME=jammy 2024-03-19T01:09:15.673 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-03-19T01:09:15.698 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-19T01:09:15.707 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-19T01:09:15.749 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-19T01:09:15.749 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-03-19T01:09:15.751 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2024-03-19T01:09:15.756 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-19T01:09:15.762 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-19T01:09:15.762 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-03-19T01:09:15.799 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2024-03-19T01:09:15.808 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-19T01:09:15.863 INFO:teuthology.run_tasks:Running task kernel... 2024-03-19T01:09:15.886 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-19T01:09:15.886 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-19T01:09:15.887 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T01:09:15.888 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T01:09:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:09:15.893 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-19T01:09:15.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:09:15.895 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-03-19T01:09:15.941 INFO:teuthology.orchestra.run.smithi167.stdout:5.15.0-56-generic 2024-03-19T01:09:15.942 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.15.0-56-generic 2024-03-19T01:09:15.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get clean 2024-03-19T01:09:15.944 INFO:teuthology.orchestra.run.smithi007.stdout:5.15.0-56-generic 2024-03-19T01:09:15.944 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.15.0-56-generic 2024-03-19T01:09:15.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2024-03-19T01:09:16.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2024-03-19T01:09:16.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2024-03-19T01:09:16.205 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:09:16.233 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-19T01:09:16.239 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:09:16.239 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-19T01:09:16.239 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-19T01:09:16.315 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-19T01:09:16.315 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-19T01:09:16.316 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-19T01:09:16.447 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,490 kB] 2024-03-19T01:09:16.519 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [421 kB] 2024-03-19T01:09:16.596 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,490 kB] 2024-03-19T01:09:16.672 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,246 kB] 2024-03-19T01:09:16.691 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [592 kB] 2024-03-19T01:09:16.712 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [288 kB] 2024-03-19T01:09:16.743 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-19T01:09:16.743 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,605 kB] 2024-03-19T01:09:16.753 INFO:teuthology.orchestra.run.smithi167.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [421 kB] 2024-03-19T01:09:16.820 INFO:teuthology.orchestra.run.smithi167.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-19T01:09:16.820 INFO:teuthology.orchestra.run.smithi167.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [268 kB] 2024-03-19T01:09:16.843 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [225 kB] 2024-03-19T01:09:16.846 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [592 kB] 2024-03-19T01:09:16.849 INFO:teuthology.orchestra.run.smithi167.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T01:09:16.849 INFO:teuthology.orchestra.run.smithi167.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [696 kB] 2024-03-19T01:09:16.872 INFO:teuthology.orchestra.run.smithi167.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,058 kB] 2024-03-19T01:09:16.874 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [288 kB] 2024-03-19T01:09:16.888 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-19T01:09:16.889 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-19T01:09:16.893 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,531 kB] 2024-03-19T01:09:16.919 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-19T01:09:16.921 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-19T01:09:16.922 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,605 kB] 2024-03-19T01:09:16.923 INFO:teuthology.orchestra.run.smithi167.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [239 kB] 2024-03-19T01:09:16.958 INFO:teuthology.orchestra.run.smithi167.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-19T01:09:16.959 INFO:teuthology.orchestra.run.smithi167.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-19T01:09:16.960 INFO:teuthology.orchestra.run.smithi167.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-19T01:09:16.960 INFO:teuthology.orchestra.run.smithi167.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-19T01:09:16.961 INFO:teuthology.orchestra.run.smithi167.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-19T01:09:16.961 INFO:teuthology.orchestra.run.smithi167.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-19T01:09:16.963 INFO:teuthology.orchestra.run.smithi167.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-19T01:09:16.964 INFO:teuthology.orchestra.run.smithi167.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-19T01:09:16.994 INFO:teuthology.orchestra.run.smithi167.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-19T01:09:17.013 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [255 kB] 2024-03-19T01:09:17.037 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [268 kB] 2024-03-19T01:09:17.039 INFO:teuthology.orchestra.run.smithi167.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-19T01:09:17.040 INFO:teuthology.orchestra.run.smithi167.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-19T01:09:17.041 INFO:teuthology.orchestra.run.smithi167.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-19T01:09:17.041 INFO:teuthology.orchestra.run.smithi167.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-19T01:09:17.042 INFO:teuthology.orchestra.run.smithi167.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,246 kB] 2024-03-19T01:09:17.065 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T01:09:17.065 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T01:09:17.066 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-19T01:09:17.066 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,058 kB] 2024-03-19T01:09:17.093 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [696 kB] 2024-03-19T01:09:17.098 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-19T01:09:17.116 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-19T01:09:17.117 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [239 kB] 2024-03-19T01:09:17.121 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-19T01:09:17.122 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-19T01:09:17.122 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-19T01:09:17.124 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-19T01:09:17.124 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-19T01:09:17.126 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-19T01:09:17.127 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-19T01:09:17.127 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-19T01:09:17.129 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-19T01:09:17.129 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-19T01:09:17.129 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-19T01:09:17.131 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-19T01:09:17.149 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-19T01:09:17.150 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-19T01:09:17.172 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-19T01:09:17.173 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-19T01:09:17.174 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-19T01:09:17.174 INFO:teuthology.orchestra.run.smithi007.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-19T01:09:17.207 INFO:teuthology.orchestra.run.smithi167.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [225 kB] 2024-03-19T01:09:17.219 INFO:teuthology.orchestra.run.smithi167.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-19T01:09:17.220 INFO:teuthology.orchestra.run.smithi167.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-19T01:09:17.222 INFO:teuthology.orchestra.run.smithi167.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,531 kB] 2024-03-19T01:09:17.301 INFO:teuthology.orchestra.run.smithi167.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [255 kB] 2024-03-19T01:09:17.309 INFO:teuthology.orchestra.run.smithi167.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T01:09:17.309 INFO:teuthology.orchestra.run.smithi167.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-19T01:09:17.333 INFO:teuthology.orchestra.run.smithi167.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-19T01:09:17.511 INFO:teuthology.orchestra.run.smithi167.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-19T01:09:17.917 INFO:teuthology.orchestra.run.smithi167.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-19T01:09:17.923 INFO:teuthology.orchestra.run.smithi167.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-19T01:09:17.937 INFO:teuthology.orchestra.run.smithi167.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-19T01:09:17.937 INFO:teuthology.orchestra.run.smithi167.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-19T01:09:17.940 INFO:teuthology.orchestra.run.smithi167.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-19T01:09:22.360 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 12.3 MB in 2s (6,463 kB/s) 2024-03-19T01:09:22.687 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 12.3 MB in 2s (6,156 kB/s) 2024-03-19T01:09:23.709 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-03-19T01:09:23.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T01:09:23.776 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-03-19T01:09:23.949 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2024-03-19T01:09:23.949 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2024-03-19T01:09:23.961 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-03-19T01:09:23.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T01:09:24.031 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-03-19T01:09:24.101 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:09:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:09:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:09:24.102 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:09:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-19T01:09:24.103 INFO:teuthology.orchestra.run.smithi167.stdout: python2.7-minimal 2024-03-19T01:09:24.103 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:09:24.103 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2024-03-19T01:09:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T01:09:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-19T01:09:24.104 INFO:teuthology.orchestra.run.smithi167.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-19T01:09:24.105 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2024-03-19T01:09:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-19T01:09:24.105 INFO:teuthology.orchestra.run.smithi167.stdout:Recommended packages: 2024-03-19T01:09:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: thermald 2024-03-19T01:09:24.153 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2024-03-19T01:09:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T01:09:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-19T01:09:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-19T01:09:24.155 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages will be upgraded: 2024-03-19T01:09:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-19T01:09:24.209 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-03-19T01:09:24.209 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-03-19T01:09:24.212 INFO:teuthology.orchestra.run.smithi167.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T01:09:24.212 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 113 MB of archives. 2024-03-19T01:09:24.212 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-19T01:09:24.212 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-19T01:09:24.366 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:09:24.366 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:09:24.366 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:09:24.366 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:09:24.366 INFO:teuthology.orchestra.run.smithi007.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-19T01:09:24.367 INFO:teuthology.orchestra.run.smithi007.stdout: python2.7-minimal 2024-03-19T01:09:24.367 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:09:24.367 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2024-03-19T01:09:24.368 INFO:teuthology.orchestra.run.smithi007.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T01:09:24.368 INFO:teuthology.orchestra.run.smithi007.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-19T01:09:24.368 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-19T01:09:24.369 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2024-03-19T01:09:24.369 INFO:teuthology.orchestra.run.smithi007.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-19T01:09:24.369 INFO:teuthology.orchestra.run.smithi007.stdout:Recommended packages: 2024-03-19T01:09:24.369 INFO:teuthology.orchestra.run.smithi007.stdout: thermald 2024-03-19T01:09:24.404 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2024-03-19T01:09:24.404 INFO:teuthology.orchestra.run.smithi007.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T01:09:24.404 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-19T01:09:24.404 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-19T01:09:24.405 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages will be upgraded: 2024-03-19T01:09:24.406 INFO:teuthology.orchestra.run.smithi007.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-19T01:09:24.516 INFO:teuthology.orchestra.run.smithi007.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T01:09:24.516 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 113 MB of archives. 2024-03-19T01:09:24.516 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-19T01:09:24.516 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-19T01:09:24.801 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-19T01:09:25.116 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-19T01:09:25.609 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-19T01:09:25.898 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-19T01:09:26.904 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-19T01:09:26.904 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-19T01:09:26.905 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-19T01:09:27.337 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-19T01:09:27.402 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-19T01:09:27.789 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 113 MB in 3s (34.9 MB/s) 2024-03-19T01:09:27.973 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-19T01:09:27.973 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-19T01:09:27.973 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-19T01:09:27.975 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-19T01:09:28.144 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-19T01:09:28.186 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-19T01:09:28.599 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 113 MB in 4s (30.0 MB/s) 2024-03-19T01:09:28.782 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-19T01:09:30.012 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-19T01:09:30.017 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:09:30.154 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:30.850 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-19T01:09:30.854 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:09:31.097 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:32.406 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-19T01:09:32.428 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:09:32.506 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:32.994 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-19T01:09:33.014 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:09:33.043 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:33.282 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-19T01:09:33.304 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:09:33.389 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:33.836 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-19T01:09:33.856 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:09:33.894 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:40.338 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:09:40.464 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:09:40.772 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:09:40.805 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:09:40.874 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:09:40.943 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:09:41.208 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:09:41.229 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-19T01:09:41.241 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-19T01:09:41.296 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T01:09:41.336 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:09:41.647 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-19T01:09:41.670 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-19T01:09:41.714 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T01:09:47.422 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-19T01:09:47.459 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:09:47.497 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:47.530 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-19T01:09:47.568 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:09:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:50.198 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:09:50.320 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:09:50.374 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:09:50.499 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:09:50.636 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T01:09:50.783 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:50.832 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T01:09:50.909 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-19T01:09:50.954 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:51.063 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:51.080 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-19T01:09:51.208 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:52.364 INFO:teuthology.orchestra.run.smithi167.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-19T01:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:09:52.365 INFO:teuthology.orchestra.run.smithi167.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-19T01:09:52.566 INFO:teuthology.orchestra.run.smithi007.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:09:52.567 INFO:teuthology.orchestra.run.smithi007.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-19T01:09:52.567 INFO:teuthology.orchestra.run.smithi007.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:09:52.567 INFO:teuthology.orchestra.run.smithi007.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-19T01:09:52.625 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:52.728 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:54.042 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:54.161 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:55.468 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-19T01:09:55.588 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-19T01:09:55.594 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-19T01:09:55.714 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-19T01:09:55.737 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:55.805 INFO:teuthology.orchestra.run.smithi167.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-19T01:09:55.805 INFO:teuthology.orchestra.run.smithi167.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:09:55.840 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:09:55.902 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-19T01:09:55.902 INFO:teuthology.orchestra.run.smithi007.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:10:10.332 INFO:teuthology.orchestra.run.smithi167.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-19T01:10:10.332 INFO:teuthology.orchestra.run.smithi167.stdout:Sourcing file `/etc/default/grub' 2024-03-19T01:10:10.364 INFO:teuthology.orchestra.run.smithi167.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T01:10:10.420 INFO:teuthology.orchestra.run.smithi167.stdout:Generating grub configuration file ... 2024-03-19T01:10:10.699 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-19T01:10:10.699 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub' 2024-03-19T01:10:10.733 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T01:10:10.754 INFO:teuthology.orchestra.run.smithi007.stdout:Generating grub configuration file ... 2024-03-19T01:10:10.760 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:10:10.784 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:10:10.993 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:10:11.017 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:10:11.223 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:10:11.229 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T01:10:11.304 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T01:10:11.306 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:10:11.310 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T01:10:11.311 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T01:10:11.388 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T01:10:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T01:10:11.449 INFO:teuthology.orchestra.run.smithi167.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T01:10:11.449 INFO:teuthology.orchestra.run.smithi167.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T01:10:11.449 INFO:teuthology.orchestra.run.smithi167.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T01:10:11.493 INFO:teuthology.orchestra.run.smithi167.stdout:done 2024-03-19T01:10:11.559 INFO:teuthology.orchestra.run.smithi007.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T01:10:11.559 INFO:teuthology.orchestra.run.smithi007.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T01:10:11.559 INFO:teuthology.orchestra.run.smithi007.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T01:10:11.592 INFO:teuthology.orchestra.run.smithi007.stdout:done 2024-03-19T01:10:12.005 DEBUG:teuthology.orchestra.run.smithi167:> dpkg -s linux-image-generic 2024-03-19T01:10:12.025 INFO:teuthology.orchestra.run.smithi167.stdout:Package: linux-image-generic 2024-03-19T01:10:12.025 INFO:teuthology.orchestra.run.smithi167.stdout:Status: install ok installed 2024-03-19T01:10:12.025 INFO:teuthology.orchestra.run.smithi167.stdout:Priority: optional 2024-03-19T01:10:12.025 INFO:teuthology.orchestra.run.smithi167.stdout:Section: kernel 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout:Installed-Size: 21 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: amd64 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout:Source: linux-meta 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout:Version: 5.15.0.101.98 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout:Recommends: thermald 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout:Description: Generic Linux kernel image 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: This package will always depend on the latest generic kernel image 2024-03-19T01:10:12.026 INFO:teuthology.orchestra.run.smithi167.stdout: available. 2024-03-19T01:10:12.026 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-19T01:10:12.027 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T01:10:12.027 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T01:10:12.027 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-19T01:10:12.027 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T01:10:12.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get clean 2024-03-19T01:10:12.081 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic 2024-03-19T01:10:12.111 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 21 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.15.0.101.98 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2024-03-19T01:10:12.112 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2024-03-19T01:10:12.113 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2024-03-19T01:10:12.113 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-19T01:10:12.113 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T01:10:12.113 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T01:10:12.113 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-19T01:10:12.113 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T01:10:12.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2024-03-19T01:10:12.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2024-03-19T01:10:12.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2024-03-19T01:10:12.367 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T01:10:12.393 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:10:12.404 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T01:10:12.424 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T01:10:12.481 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:10:12.508 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T01:10:12.561 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T01:10:12.643 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T01:10:13.772 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-03-19T01:10:13.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T01:10:13.843 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-03-19T01:10:14.020 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-03-19T01:10:14.021 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-03-19T01:10:14.177 INFO:teuthology.orchestra.run.smithi007.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-19T01:10:14.177 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:10:14.177 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:10:14.178 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:10:14.178 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:10:14.178 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:10:14.178 INFO:teuthology.orchestra.run.smithi007.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:10:14.178 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:10:14.217 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T01:10:14.219 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic 2024-03-19T01:10:14.280 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 21 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.15.0.101.98 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2024-03-19T01:10:14.281 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2024-03-19T01:10:14.282 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2024-03-19T01:10:14.282 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2024-03-19T01:10:14.282 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-19T01:10:14.331 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.iXbgbwIyKd 2024-03-19T01:10:14.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub/grub.cfg /tmp/tmp.iXbgbwIyKd 2024-03-19T01:10:14.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.iXbgbwIyKd 2024-03-19T01:10:14.458 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-03-19T01:10:14.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T01:10:14.497 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.iXbgbwIyKd is 10KB 2024-03-19T01:10:14.510 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.iXbgbwIyKd 2024-03-19T01:10:14.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-19T01:10:14.555 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-03-19T01:10:14.569 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:10:14.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-19T01:10:14.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-19T01:10:14.637 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-19T01:10:14.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo update-grub 2024-03-19T01:10:14.752 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2024-03-19T01:10:14.752 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2024-03-19T01:10:14.917 INFO:teuthology.orchestra.run.smithi167.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-19T01:10:14.918 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:10:14.918 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:10:14.918 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:10:14.918 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:10:14.918 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:10:14.918 INFO:teuthology.orchestra.run.smithi167.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:10:14.919 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:10:14.956 INFO:teuthology.orchestra.run.smithi167.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T01:10:14.958 DEBUG:teuthology.orchestra.run.smithi167:> dpkg -s linux-image-generic 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Package: linux-image-generic 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Status: install ok installed 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Priority: optional 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Section: kernel 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Installed-Size: 21 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: amd64 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Source: linux-meta 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Version: 5.15.0.101.98 2024-03-19T01:10:14.970 INFO:teuthology.orchestra.run.smithi167.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T01:10:14.971 INFO:teuthology.orchestra.run.smithi167.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T01:10:14.971 INFO:teuthology.orchestra.run.smithi167.stdout:Recommends: thermald 2024-03-19T01:10:14.971 INFO:teuthology.orchestra.run.smithi167.stdout:Description: Generic Linux kernel image 2024-03-19T01:10:14.971 INFO:teuthology.orchestra.run.smithi167.stdout: This package will always depend on the latest generic kernel image 2024-03-19T01:10:14.971 INFO:teuthology.orchestra.run.smithi167.stdout: available. 2024-03-19T01:10:14.972 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-03-19T01:10:15.016 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.UGnMm50AJw 2024-03-19T01:10:15.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub/grub.cfg /tmp/tmp.UGnMm50AJw 2024-03-19T01:10:15.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.UGnMm50AJw 2024-03-19T01:10:15.187 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.UGnMm50AJw is 10KB 2024-03-19T01:10:15.199 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.UGnMm50AJw 2024-03-19T01:10:15.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-19T01:10:15.259 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:10:15.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-19T01:10:15.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-19T01:10:15.350 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-19T01:10:15.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo update-grub 2024-03-19T01:10:15.840 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub' 2024-03-19T01:10:15.842 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T01:10:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-03-19T01:10:16.014 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:10:16.020 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:10:16.184 INFO:teuthology.orchestra.run.smithi167.stderr:Sourcing file `/etc/default/grub' 2024-03-19T01:10:16.185 INFO:teuthology.orchestra.run.smithi167.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T01:10:16.188 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2024-03-19T01:10:16.240 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:10:16.245 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T01:10:16.287 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T01:10:16.293 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T01:10:16.377 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:10:16.382 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:10:16.410 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T01:10:16.410 INFO:teuthology.orchestra.run.smithi007.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T01:10:16.410 INFO:teuthology.orchestra.run.smithi007.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T01:10:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-03-19T01:10:16.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-03-19T01:10:16.645 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:10:16.650 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T01:10:16.691 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T01:10:16.696 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T01:10:16.875 INFO:teuthology.orchestra.run.smithi167.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T01:10:16.875 INFO:teuthology.orchestra.run.smithi167.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T01:10:16.875 INFO:teuthology.orchestra.run.smithi167.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T01:10:16.907 INFO:teuthology.orchestra.run.smithi167.stderr:done 2024-03-19T01:10:16.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2024-03-19T01:10:46.471 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-19T01:10:46.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-19T01:10:46.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:10:46.913 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-19T01:10:46.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-19T01:10:46.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:11:21.379 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-03-19T01:11:30.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-19T01:11:30.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:11:46.490 DEBUG:teuthology.orchestra.remote:timed out 2024-03-19T01:11:48.770 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-03-19T01:11:55.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-19T01:11:55.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:11:56.271 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-19T01:11:57.010 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-19T01:11:57.010 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-03-19T01:11:57.010 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-19T01:11:57.056 INFO:teuthology.orchestra.run.smithi007.stdout:5.15.0-101-generic 2024-03-19T01:11:57.056 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-19T01:11:57.056 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T01:11:57.056 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-19T01:11:58.057 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-19T01:11:58.057 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T01:11:58.143 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-03-19T01:11:58.161 DEBUG:teuthology.parallel:result is None 2024-03-19T01:12:00.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-19T01:12:00.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:12:01.677 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-03-19T01:12:16.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-19T01:12:16.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:12:16.941 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-03-19T01:12:17.514 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-19T01:12:17.514 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-101-generic"... 2024-03-19T01:12:17.515 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-03-19T01:12:17.562 INFO:teuthology.orchestra.run.smithi167.stdout:5.15.0-101-generic 2024-03-19T01:12:17.563 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-19T01:12:17.563 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T01:12:17.563 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-19T01:12:18.564 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-19T01:12:18.564 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T01:12:18.645 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2024-03-19T01:12:18.672 DEBUG:teuthology.parallel:result is None 2024-03-19T01:12:18.673 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-19T01:12:18.681 INFO:teuthology.task.internal:Creating test directory... 2024-03-19T01:12:18.682 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T01:12:18.684 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T01:12:18.689 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-19T01:12:18.812 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-19T01:12:18.861 INFO:teuthology.task.internal:Creating archive directory... 2024-03-19T01:12:18.861 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T01:12:18.864 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T01:12:18.881 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-19T01:12:18.888 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-19T01:12:18.888 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T01:12:18.913 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T01:12:18.929 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:12:18.934 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:12:18.937 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:12:18.942 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:12:18.943 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-19T01:12:18.951 INFO:teuthology.task.internal:Configuring sudo... 2024-03-19T01:12:18.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T01:12:18.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T01:12:18.997 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-19T01:12:19.029 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-19T01:12:19.029 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T01:12:19.041 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T01:12:19.045 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:12:19.093 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:12:19.145 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:12:19.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T01:12:19.202 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:12:19.208 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:12:19.255 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:12:19.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T01:12:19.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-03-19T01:12:19.318 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2024-03-19T01:12:19.389 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-19T01:12:19.399 INFO:teuthology.task.internal:Starting timer... 2024-03-19T01:12:19.399 INFO:teuthology.run_tasks:Running task pcp... 2024-03-19T01:12:19.420 INFO:teuthology.run_tasks:Running task selinux... 2024-03-19T01:12:19.493 DEBUG:teuthology.task.selinux:Excluding smithi007: OS 'ubuntu' does not support SELinux 2024-03-19T01:12:19.493 DEBUG:teuthology.task.selinux:Excluding smithi167: OS 'ubuntu' does not support SELinux 2024-03-19T01:12:19.493 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-19T01:12:19.493 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-19T01:12:19.493 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-19T01:12:19.493 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-19T01:12:19.541 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-19T01:12:19.563 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-19T01:12:19.564 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-19T01:19:22.692 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2024-03-19T01:19:22.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-19T01:19:22.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:19:22.805 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-19T01:19:22.883 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-19T01:19:22.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-19T01:19:22.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:19:22.964 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-03-19T01:19:23.042 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-19T01:19:23.042 INFO:teuthology.run_tasks:Running task clock... 2024-03-19T01:19:23.053 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-19T01:19:23.054 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T01:19:23.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T01:19:23.057 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T01:19:23.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T01:19:23.086 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-19T01:19:23.086 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: Command line: ntpd -gq 2024-03-19T01:19:23.087 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: ---------------------------------------------------- 2024-03-19T01:19:23.087 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: ntp-4 is maintained by Network Time Foundation, 2024-03-19T01:19:23.087 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-19T01:19:23.087 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: corporation. Support and training for ntp-4 are 2024-03-19T01:19:23.087 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: available at https://www.nwtime.org/support 2024-03-19T01:19:23.087 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: ---------------------------------------------------- 2024-03-19T01:19:23.088 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: proto: precision = 0.068 usec (-24) 2024-03-19T01:19:23.088 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: basedate set to 2022-02-04 2024-03-19T01:19:23.088 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: gps base set to 2022-02-06 (week 2196) 2024-03-19T01:19:23.089 INFO:teuthology.orchestra.run.smithi007.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T01:19:23.089 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T01:19:23.089 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: restrict ::: KOD does nothing without LIMITED. 2024-03-19T01:19:23.090 INFO:teuthology.orchestra.run.smithi007.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-19T01:19:23.090 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: Listen and drop on 0 v6wildcard [::]:123 2024-03-19T01:19:23.090 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-19T01:19:23.091 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: Listen normally on 2 lo 127.0.0.1:123 2024-03-19T01:19:23.091 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: Listen normally on 3 ens1f0 172.21.15.7:123 2024-03-19T01:19:23.091 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: Listen normally on 4 lo [::1]:123 2024-03-19T01:19:23.091 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15ea%3]:123 2024-03-19T01:19:23.091 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:23 ntpd[17126]: Listening on routing socket on fd #22 for interface updates 2024-03-19T01:19:23.110 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-19T01:19:23.110 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: Command line: ntpd -gq 2024-03-19T01:19:23.110 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: ---------------------------------------------------- 2024-03-19T01:19:23.110 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: ntp-4 is maintained by Network Time Foundation, 2024-03-19T01:19:23.110 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-19T01:19:23.111 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: corporation. Support and training for ntp-4 are 2024-03-19T01:19:23.111 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: available at https://www.nwtime.org/support 2024-03-19T01:19:23.111 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: ---------------------------------------------------- 2024-03-19T01:19:23.111 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: proto: precision = 0.060 usec (-24) 2024-03-19T01:19:23.111 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: basedate set to 2022-02-04 2024-03-19T01:19:23.111 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: gps base set to 2022-02-06 (week 2196) 2024-03-19T01:19:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T01:19:23.112 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T01:19:23.112 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: restrict ::: KOD does nothing without LIMITED. 2024-03-19T01:19:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-19T01:19:23.113 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: Listen and drop on 0 v6wildcard [::]:123 2024-03-19T01:19:23.113 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-19T01:19:23.113 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: Listen normally on 2 lo 127.0.0.1:123 2024-03-19T01:19:23.114 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: Listen normally on 3 enp3s0f1 172.21.15.167:123 2024-03-19T01:19:23.114 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: Listen normally on 4 lo [::1]:123 2024-03-19T01:19:23.114 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7309%5]:123 2024-03-19T01:19:23.114 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:23 ntpd[17195]: Listening on routing socket on fd #22 for interface updates 2024-03-19T01:19:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:24 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:24 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:24 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:24 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:24 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:24 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:24 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:24 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:24.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:24.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:24 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:25.089 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:25 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:25.090 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:25 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:25.090 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:25 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:25.090 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:25 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:25.090 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:25 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:25.090 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:25 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:25.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:25 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:25.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:25 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:25.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:25 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:25.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:25 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:25.113 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:25 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:25.113 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:25 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:26.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:26 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:26.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:26 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:26.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:26 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:26.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:26 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:26.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:26 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:26.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:26 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:26.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:26.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:26.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:26 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:27 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:27 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:27 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:27 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:27.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:27 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:27.089 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:27 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:27.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:27 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:27.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:27 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:27.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:27 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:27.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:27 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:27.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:27 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:27.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:27 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:28.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:28.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:28.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:28.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:28.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:28.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:28.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:28.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:28.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:28.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:28.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:28.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:28 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:28.113 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:28.113 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:28.113 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:28 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:29 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:29 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:29 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:29 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:29 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:29 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:29.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:29 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:29.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:29 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:29.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:29 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:29.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:29 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:29.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:29 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:29.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:29 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:30.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:30.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:30.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:30.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:30.087 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:30.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:30.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:30.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:30.088 INFO:teuthology.orchestra.run.smithi007.stdout:19 Mar 01:19:30 ntpd[17126]: ntpd: time slew +0.001026 s 2024-03-19T01:19:30.088 INFO:teuthology.orchestra.run.smithi007.stdout:ntpd: time slew +0.001026s 2024-03-19T01:19:30.088 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-19T01:19:30.089 INFO:teuthology.orchestra.run.smithi007.stderr:19 Mar 01:19:30 ntpd[17126]: can't open /var/log/ntpstats/loopstats.20240319: Permission denied 2024-03-19T01:19:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:19:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:19:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:19:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:19:30.112 INFO:teuthology.orchestra.run.smithi167.stdout:19 Mar 01:19:30 ntpd[17195]: ntpd: time slew +0.000226 s 2024-03-19T01:19:30.112 INFO:teuthology.orchestra.run.smithi167.stdout:ntpd: time slew +0.000226s 2024-03-19T01:19:30.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-19T01:19:30.112 INFO:teuthology.orchestra.run.smithi167.stderr:19 Mar 01:19:30 ntpd[17195]: can't open /var/log/ntpstats/loopstats.20240319: Permission denied 2024-03-19T01:19:30.141 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T01:19:30.141 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2024-03-19T01:19:30.141 INFO:teuthology.orchestra.run.smithi007.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:19:30.141 INFO:teuthology.orchestra.run.smithi007.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:19:30.141 INFO:teuthology.orchestra.run.smithi007.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:19:30.141 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:19:30.175 INFO:teuthology.orchestra.run.smithi167.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T01:19:30.175 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================== 2024-03-19T01:19:30.175 INFO:teuthology.orchestra.run.smithi167.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:19:30.175 INFO:teuthology.orchestra.run.smithi167.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:19:30.175 INFO:teuthology.orchestra.run.smithi167.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:19:30.175 INFO:teuthology.orchestra.run.smithi167.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:19:30.175 INFO:teuthology.run_tasks:Running task install... 2024-03-19T01:19:30.184 DEBUG:teuthology.task.install:project ceph 2024-03-19T01:19:30.184 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}} 2024-03-19T01:19:30.185 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-19T01:19:30.185 INFO:teuthology.task.install:Using flavor: default 2024-03-19T01:19:30.190 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-19T01:19:30.190 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-19T01:19:30.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-key list | grep Ceph 2024-03-19T01:19:30.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-key list | grep Ceph 2024-03-19T01:19:30.253 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-19T01:19:30.273 INFO:teuthology.orchestra.run.smithi167.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-19T01:19:30.283 INFO:teuthology.orchestra.run.smithi007.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-19T01:19:30.283 INFO:teuthology.orchestra.run.smithi007.stdout:uid [ unknown] Ceph.com (release key) 2024-03-19T01:19:30.284 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-19T01:19:30.284 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-19T01:19:30.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:19:30.302 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-19T01:19:30.302 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph.com (release key) 2024-03-19T01:19:30.302 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-19T01:19:30.303 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-19T01:19:30.303 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:19:30.429 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default/ 2024-03-19T01:19:30.429 INFO:teuthology.task.install.deb:Package version is 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:19:30.517 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default/ 2024-03-19T01:19:30.517 INFO:teuthology.task.install.deb:Package version is 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:19:30.827 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:19:30.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-19T01:19:30.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2024-03-19T01:19:30.853 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:19:30.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-19T01:19:30.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2024-03-19T01:19:31.017 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:19:31.017 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T01:19:31.028 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T01:19:31.049 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T01:19:31.160 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:19:31.160 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T01:19:31.232 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T01:19:31.314 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T01:19:31.357 INFO:teuthology.orchestra.run.smithi007.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy InRelease 2024-03-19T01:19:31.378 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy InRelease 2024-03-19T01:19:31.453 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-03-19T01:19:31.474 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-03-19T01:19:31.548 INFO:teuthology.orchestra.run.smithi007.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-19T01:19:31.566 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-19T01:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-19T01:19:31.657 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-19T01:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-19T01:19:31.758 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-19T01:19:31.839 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 28.2 kB in 1s (35.6 kB/s) 2024-03-19T01:19:31.848 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 28.2 kB in 1s (35.5 kB/s) 2024-03-19T01:19:32.811 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-03-19T01:19:32.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2206-gb6ae2491-1jammy cephadm=19.0.0-2206-gb6ae2491-1jammy ceph-mds=19.0.0-2206-gb6ae2491-1jammy ceph-mgr=19.0.0-2206-gb6ae2491-1jammy ceph-common=19.0.0-2206-gb6ae2491-1jammy ceph-fuse=19.0.0-2206-gb6ae2491-1jammy ceph-test=19.0.0-2206-gb6ae2491-1jammy ceph-volume=19.0.0-2206-gb6ae2491-1jammy radosgw=19.0.0-2206-gb6ae2491-1jammy python3-rados=19.0.0-2206-gb6ae2491-1jammy python3-rgw=19.0.0-2206-gb6ae2491-1jammy python3-cephfs=19.0.0-2206-gb6ae2491-1jammy python3-rbd=19.0.0-2206-gb6ae2491-1jammy libcephfs2=19.0.0-2206-gb6ae2491-1jammy libcephfs-dev=19.0.0-2206-gb6ae2491-1jammy librados2=19.0.0-2206-gb6ae2491-1jammy librbd1=19.0.0-2206-gb6ae2491-1jammy rbd-fuse=19.0.0-2206-gb6ae2491-1jammy python3-cephfs=19.0.0-2206-gb6ae2491-1jammy cephfs-shell=19.0.0-2206-gb6ae2491-1jammy cephfs-top=19.0.0-2206-gb6ae2491-1jammy cephfs-mirror=19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:19:32.843 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-03-19T01:19:32.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2206-gb6ae2491-1jammy cephadm=19.0.0-2206-gb6ae2491-1jammy ceph-mds=19.0.0-2206-gb6ae2491-1jammy ceph-mgr=19.0.0-2206-gb6ae2491-1jammy ceph-common=19.0.0-2206-gb6ae2491-1jammy ceph-fuse=19.0.0-2206-gb6ae2491-1jammy ceph-test=19.0.0-2206-gb6ae2491-1jammy ceph-volume=19.0.0-2206-gb6ae2491-1jammy radosgw=19.0.0-2206-gb6ae2491-1jammy python3-rados=19.0.0-2206-gb6ae2491-1jammy python3-rgw=19.0.0-2206-gb6ae2491-1jammy python3-cephfs=19.0.0-2206-gb6ae2491-1jammy python3-rbd=19.0.0-2206-gb6ae2491-1jammy libcephfs2=19.0.0-2206-gb6ae2491-1jammy libcephfs-dev=19.0.0-2206-gb6ae2491-1jammy librados2=19.0.0-2206-gb6ae2491-1jammy librbd1=19.0.0-2206-gb6ae2491-1jammy rbd-fuse=19.0.0-2206-gb6ae2491-1jammy python3-cephfs=19.0.0-2206-gb6ae2491-1jammy cephfs-shell=19.0.0-2206-gb6ae2491-1jammy cephfs-top=19.0.0-2206-gb6ae2491-1jammy cephfs-mirror=19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:19:32.891 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-03-19T01:19:32.919 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-03-19T01:19:33.084 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-03-19T01:19:33.085 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-03-19T01:19:33.114 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2024-03-19T01:19:33.114 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2024-03-19T01:19:33.271 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:19:33.271 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:19:33.271 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:19:33.271 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:19:33.271 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:19:33.271 INFO:teuthology.orchestra.run.smithi007.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:19:33.271 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:19:33.273 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-19T01:19:33.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-19T01:19:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout:Recommended packages: 2024-03-19T01:19:33.276 INFO:teuthology.orchestra.run.smithi007.stdout: btrfs-tools 2024-03-19T01:19:33.283 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:19:33.284 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:19:33.284 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:19:33.284 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:19:33.284 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:19:33.284 INFO:teuthology.orchestra.run.smithi167.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:19:33.284 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:19:33.285 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2024-03-19T01:19:33.285 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-19T01:19:33.285 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-19T01:19:33.285 INFO:teuthology.orchestra.run.smithi167.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-19T01:19:33.285 INFO:teuthology.orchestra.run.smithi167.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-19T01:19:33.286 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-19T01:19:33.287 INFO:teuthology.orchestra.run.smithi167.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-19T01:19:33.288 INFO:teuthology.orchestra.run.smithi167.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-19T01:19:33.288 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-19T01:19:33.288 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-19T01:19:33.288 INFO:teuthology.orchestra.run.smithi167.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-19T01:19:33.288 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2024-03-19T01:19:33.288 INFO:teuthology.orchestra.run.smithi167.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-19T01:19:33.288 INFO:teuthology.orchestra.run.smithi167.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-19T01:19:33.288 INFO:teuthology.orchestra.run.smithi167.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-19T01:19:33.288 INFO:teuthology.orchestra.run.smithi167.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-19T01:19:33.289 INFO:teuthology.orchestra.run.smithi167.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-19T01:19:33.289 INFO:teuthology.orchestra.run.smithi167.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-19T01:19:33.289 INFO:teuthology.orchestra.run.smithi167.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-19T01:19:33.289 INFO:teuthology.orchestra.run.smithi167.stdout:Recommended packages: 2024-03-19T01:19:33.289 INFO:teuthology.orchestra.run.smithi167.stdout: btrfs-tools 2024-03-19T01:19:33.389 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2024-03-19T01:19:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-19T01:19:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-19T01:19:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-19T01:19:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-19T01:19:33.389 INFO:teuthology.orchestra.run.smithi007.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-19T01:19:33.390 INFO:teuthology.orchestra.run.smithi007.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-19T01:19:33.390 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-19T01:19:33.391 INFO:teuthology.orchestra.run.smithi007.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-19T01:19:33.391 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-19T01:19:33.391 INFO:teuthology.orchestra.run.smithi007.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-19T01:19:33.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-19T01:19:33.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-19T01:19:33.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-19T01:19:33.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-19T01:19:33.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-19T01:19:33.392 INFO:teuthology.orchestra.run.smithi007.stdout: socat unzip xmlstarlet zip 2024-03-19T01:19:33.394 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2024-03-19T01:19:33.394 INFO:teuthology.orchestra.run.smithi167.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-19T01:19:33.394 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-19T01:19:33.394 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-19T01:19:33.394 INFO:teuthology.orchestra.run.smithi167.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-19T01:19:33.394 INFO:teuthology.orchestra.run.smithi167.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-19T01:19:33.395 INFO:teuthology.orchestra.run.smithi167.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-19T01:19:33.396 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-19T01:19:33.396 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-19T01:19:33.396 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-19T01:19:33.396 INFO:teuthology.orchestra.run.smithi167.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-19T01:19:33.396 INFO:teuthology.orchestra.run.smithi167.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-19T01:19:33.396 INFO:teuthology.orchestra.run.smithi167.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-19T01:19:33.396 INFO:teuthology.orchestra.run.smithi167.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-19T01:19:33.396 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-19T01:19:33.396 INFO:teuthology.orchestra.run.smithi167.stdout: socat unzip xmlstarlet zip 2024-03-19T01:19:33.458 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 124 newly installed, 0 to remove and 285 not upgraded. 2024-03-19T01:19:33.458 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 178 MB of archives. 2024-03-19T01:19:33.458 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 739 MB of additional disk space will be used. 2024-03-19T01:19:33.458 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-19T01:19:33.480 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-19T01:19:33.491 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-19T01:19:33.540 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-19T01:19:33.565 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-19T01:19:33.569 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-19T01:19:33.582 INFO:teuthology.orchestra.run.smithi167.stdout:0 upgraded, 124 newly installed, 0 to remove and 285 not upgraded. 2024-03-19T01:19:33.582 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 178 MB of archives. 2024-03-19T01:19:33.582 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 739 MB of additional disk space will be used. 2024-03-19T01:19:33.582 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-19T01:19:33.582 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-19T01:19:33.644 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-19T01:19:33.648 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-19T01:19:33.660 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-19T01:19:33.665 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-19T01:19:33.665 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-19T01:19:33.666 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-19T01:19:33.674 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-19T01:19:33.677 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-19T01:19:33.679 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-19T01:19:33.680 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-19T01:19:33.680 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-19T01:19:33.691 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-19T01:19:33.693 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-19T01:19:33.695 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-19T01:19:33.696 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-19T01:19:33.711 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-19T01:19:33.712 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-19T01:19:33.713 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-19T01:19:33.713 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-19T01:19:33.714 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-19T01:19:33.714 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-19T01:19:33.715 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-19T01:19:33.733 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-19T01:19:33.733 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-19T01:19:33.736 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-19T01:19:33.736 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-19T01:19:33.753 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-19T01:19:33.754 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-19T01:19:33.755 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-19T01:19:33.755 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-19T01:19:33.755 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-19T01:19:33.757 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-19T01:19:33.774 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-19T01:19:33.775 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-19T01:19:33.778 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2206-gb6ae2491-1jammy [3,546 kB] 2024-03-19T01:19:33.780 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-19T01:19:33.780 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-19T01:19:33.795 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-19T01:19:33.795 INFO:teuthology.orchestra.run.smithi007.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-19T01:19:33.796 INFO:teuthology.orchestra.run.smithi007.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-19T01:19:33.799 INFO:teuthology.orchestra.run.smithi007.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-19T01:19:33.799 INFO:teuthology.orchestra.run.smithi007.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-19T01:19:33.801 INFO:teuthology.orchestra.run.smithi007.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-19T01:19:33.815 INFO:teuthology.orchestra.run.smithi007.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-19T01:19:33.818 INFO:teuthology.orchestra.run.smithi007.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-19T01:19:33.818 INFO:teuthology.orchestra.run.smithi007.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-19T01:19:33.819 INFO:teuthology.orchestra.run.smithi007.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-19T01:19:33.836 INFO:teuthology.orchestra.run.smithi007.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-19T01:19:33.869 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2206-gb6ae2491-1jammy [3,546 kB] 2024-03-19T01:19:33.890 INFO:teuthology.orchestra.run.smithi007.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-19T01:19:33.891 INFO:teuthology.orchestra.run.smithi007.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-19T01:19:33.893 INFO:teuthology.orchestra.run.smithi007.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-19T01:19:33.893 INFO:teuthology.orchestra.run.smithi007.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-19T01:19:33.894 INFO:teuthology.orchestra.run.smithi007.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-19T01:19:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-19T01:19:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-19T01:19:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-19T01:19:33.915 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-19T01:19:33.930 INFO:teuthology.orchestra.run.smithi007.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-19T01:19:33.930 INFO:teuthology.orchestra.run.smithi007.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-19T01:19:33.937 INFO:teuthology.orchestra.run.smithi007.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-19T01:19:33.938 INFO:teuthology.orchestra.run.smithi007.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-19T01:19:33.938 INFO:teuthology.orchestra.run.smithi007.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-19T01:19:33.939 INFO:teuthology.orchestra.run.smithi007.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-19T01:19:33.939 INFO:teuthology.orchestra.run.smithi007.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-19T01:19:33.939 INFO:teuthology.orchestra.run.smithi007.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-19T01:19:33.958 INFO:teuthology.orchestra.run.smithi007.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-19T01:19:33.960 INFO:teuthology.orchestra.run.smithi007.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-19T01:19:33.961 INFO:teuthology.orchestra.run.smithi007.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-19T01:19:33.961 INFO:teuthology.orchestra.run.smithi007.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-19T01:19:33.989 INFO:teuthology.orchestra.run.smithi007.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-19T01:19:33.992 INFO:teuthology.orchestra.run.smithi007.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-19T01:19:33.993 INFO:teuthology.orchestra.run.smithi007.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-19T01:19:33.994 INFO:teuthology.orchestra.run.smithi007.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-19T01:19:33.994 INFO:teuthology.orchestra.run.smithi007.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-19T01:19:34.000 INFO:teuthology.orchestra.run.smithi007.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-19T01:19:34.010 INFO:teuthology.orchestra.run.smithi007.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-19T01:19:34.010 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-19T01:19:34.013 INFO:teuthology.orchestra.run.smithi007.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-19T01:19:34.014 INFO:teuthology.orchestra.run.smithi007.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-19T01:19:34.014 INFO:teuthology.orchestra.run.smithi007.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-19T01:19:34.024 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-19T01:19:34.034 INFO:teuthology.orchestra.run.smithi007.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-19T01:19:34.034 INFO:teuthology.orchestra.run.smithi007.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-19T01:19:34.034 INFO:teuthology.orchestra.run.smithi007.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-19T01:19:34.042 INFO:teuthology.orchestra.run.smithi007.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-19T01:19:34.042 INFO:teuthology.orchestra.run.smithi007.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-19T01:19:34.042 INFO:teuthology.orchestra.run.smithi007.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-19T01:19:34.052 INFO:teuthology.orchestra.run.smithi007.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-19T01:19:34.053 INFO:teuthology.orchestra.run.smithi007.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-19T01:19:34.077 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-19T01:19:34.253 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-19T01:19:34.263 INFO:teuthology.orchestra.run.smithi167.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-19T01:19:34.290 INFO:teuthology.orchestra.run.smithi167.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-19T01:19:34.299 INFO:teuthology.orchestra.run.smithi167.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-19T01:19:34.464 INFO:teuthology.orchestra.run.smithi167.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-19T01:19:35.032 INFO:teuthology.orchestra.run.smithi167.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-19T01:19:35.047 INFO:teuthology.orchestra.run.smithi167.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-19T01:19:35.063 INFO:teuthology.orchestra.run.smithi167.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-19T01:19:35.072 INFO:teuthology.orchestra.run.smithi167.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-19T01:19:35.072 INFO:teuthology.orchestra.run.smithi167.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-19T01:19:35.089 INFO:teuthology.orchestra.run.smithi167.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-19T01:19:35.126 INFO:teuthology.orchestra.run.smithi167.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-19T01:19:35.139 INFO:teuthology.orchestra.run.smithi167.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-19T01:19:35.144 INFO:teuthology.orchestra.run.smithi167.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-19T01:19:35.144 INFO:teuthology.orchestra.run.smithi167.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-19T01:19:35.218 INFO:teuthology.orchestra.run.smithi007.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2206-gb6ae2491-1jammy [3,225 kB] 2024-03-19T01:19:35.316 INFO:teuthology.orchestra.run.smithi167.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-19T01:19:35.399 INFO:teuthology.orchestra.run.smithi167.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-19T01:19:35.415 INFO:teuthology.orchestra.run.smithi167.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-19T01:19:35.481 INFO:teuthology.orchestra.run.smithi167.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-19T01:19:35.510 INFO:teuthology.orchestra.run.smithi167.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-19T01:19:35.529 INFO:teuthology.orchestra.run.smithi167.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-19T01:19:35.554 INFO:teuthology.orchestra.run.smithi167.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-19T01:19:35.585 INFO:teuthology.orchestra.run.smithi167.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2206-gb6ae2491-1jammy [3,225 kB] 2024-03-19T01:19:35.737 INFO:teuthology.orchestra.run.smithi167.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-19T01:19:35.751 INFO:teuthology.orchestra.run.smithi167.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-19T01:19:35.758 INFO:teuthology.orchestra.run.smithi167.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-19T01:19:35.763 INFO:teuthology.orchestra.run.smithi167.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-19T01:19:35.947 INFO:teuthology.orchestra.run.smithi167.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-19T01:19:36.031 INFO:teuthology.orchestra.run.smithi167.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-19T01:19:36.046 INFO:teuthology.orchestra.run.smithi167.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-19T01:19:36.198 INFO:teuthology.orchestra.run.smithi007.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2206-gb6ae2491-1jammy [712 kB] 2024-03-19T01:19:36.230 INFO:teuthology.orchestra.run.smithi167.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-19T01:19:36.272 INFO:teuthology.orchestra.run.smithi167.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2206-gb6ae2491-1jammy [712 kB] 2024-03-19T01:19:36.281 INFO:teuthology.orchestra.run.smithi167.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-19T01:19:36.295 INFO:teuthology.orchestra.run.smithi167.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-19T01:19:36.374 INFO:teuthology.orchestra.run.smithi007.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2206-gb6ae2491-1jammy [356 kB] 2024-03-19T01:19:36.429 INFO:teuthology.orchestra.run.smithi167.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2206-gb6ae2491-1jammy [356 kB] 2024-03-19T01:19:36.454 INFO:teuthology.orchestra.run.smithi167.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-19T01:19:36.454 INFO:teuthology.orchestra.run.smithi167.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-19T01:19:36.459 INFO:teuthology.orchestra.run.smithi007.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2206-gb6ae2491-1jammy [32.8 kB] 2024-03-19T01:19:36.460 INFO:teuthology.orchestra.run.smithi167.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-19T01:19:36.464 INFO:teuthology.orchestra.run.smithi167.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-19T01:19:36.479 INFO:teuthology.orchestra.run.smithi167.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-19T01:19:36.503 INFO:teuthology.orchestra.run.smithi007.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2206-gb6ae2491-1jammy [180 kB] 2024-03-19T01:19:36.505 INFO:teuthology.orchestra.run.smithi167.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2206-gb6ae2491-1jammy [32.8 kB] 2024-03-19T01:19:36.507 INFO:teuthology.orchestra.run.smithi167.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2206-gb6ae2491-1jammy [180 kB] 2024-03-19T01:19:36.519 INFO:teuthology.orchestra.run.smithi167.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2206-gb6ae2491-1jammy [64.6 kB] 2024-03-19T01:19:36.522 INFO:teuthology.orchestra.run.smithi167.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2206-gb6ae2491-1jammy [330 kB] 2024-03-19T01:19:36.532 INFO:teuthology.orchestra.run.smithi007.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2206-gb6ae2491-1jammy [64.6 kB] 2024-03-19T01:19:36.575 INFO:teuthology.orchestra.run.smithi007.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2206-gb6ae2491-1jammy [330 kB] 2024-03-19T01:19:36.596 INFO:teuthology.orchestra.run.smithi167.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2206-gb6ae2491-1jammy [6,651 kB] 2024-03-19T01:19:36.653 INFO:teuthology.orchestra.run.smithi007.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2206-gb6ae2491-1jammy [6,651 kB] 2024-03-19T01:19:36.663 INFO:teuthology.orchestra.run.smithi167.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-19T01:19:36.928 INFO:teuthology.orchestra.run.smithi167.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-19T01:19:37.044 INFO:teuthology.orchestra.run.smithi167.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-19T01:19:37.089 INFO:teuthology.orchestra.run.smithi167.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-19T01:19:37.170 INFO:teuthology.orchestra.run.smithi167.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-19T01:19:37.183 INFO:teuthology.orchestra.run.smithi167.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-19T01:19:37.196 INFO:teuthology.orchestra.run.smithi167.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-19T01:19:37.236 INFO:teuthology.orchestra.run.smithi167.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-19T01:19:37.503 INFO:teuthology.orchestra.run.smithi167.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-19T01:19:37.505 INFO:teuthology.orchestra.run.smithi167.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-19T01:19:37.508 INFO:teuthology.orchestra.run.smithi167.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-19T01:19:37.596 INFO:teuthology.orchestra.run.smithi167.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2206-gb6ae2491-1jammy [112 kB] 2024-03-19T01:19:37.602 INFO:teuthology.orchestra.run.smithi167.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2206-gb6ae2491-1jammy [466 kB] 2024-03-19T01:19:37.672 INFO:teuthology.orchestra.run.smithi167.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-19T01:19:37.691 INFO:teuthology.orchestra.run.smithi167.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2206-gb6ae2491-1jammy [25.3 MB] 2024-03-19T01:19:38.008 INFO:teuthology.orchestra.run.smithi167.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-19T01:19:38.358 INFO:teuthology.orchestra.run.smithi007.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2206-gb6ae2491-1jammy [112 kB] 2024-03-19T01:19:38.368 INFO:teuthology.orchestra.run.smithi007.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2206-gb6ae2491-1jammy [466 kB] 2024-03-19T01:19:38.425 INFO:teuthology.orchestra.run.smithi167.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-19T01:19:38.435 INFO:teuthology.orchestra.run.smithi167.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-19T01:19:38.436 INFO:teuthology.orchestra.run.smithi167.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-19T01:19:38.479 INFO:teuthology.orchestra.run.smithi007.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2206-gb6ae2491-1jammy [25.3 MB] 2024-03-19T01:19:38.507 INFO:teuthology.orchestra.run.smithi167.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-19T01:19:38.508 INFO:teuthology.orchestra.run.smithi167.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-19T01:19:38.673 INFO:teuthology.orchestra.run.smithi167.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-19T01:19:38.880 INFO:teuthology.orchestra.run.smithi167.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-19T01:19:39.017 INFO:teuthology.orchestra.run.smithi167.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-19T01:19:39.152 INFO:teuthology.orchestra.run.smithi167.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-19T01:19:39.259 INFO:teuthology.orchestra.run.smithi167.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-19T01:19:39.384 INFO:teuthology.orchestra.run.smithi167.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-19T01:19:39.718 INFO:teuthology.orchestra.run.smithi167.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-19T01:19:39.835 INFO:teuthology.orchestra.run.smithi167.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-19T01:19:39.935 INFO:teuthology.orchestra.run.smithi167.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-19T01:19:40.024 INFO:teuthology.orchestra.run.smithi167.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-19T01:19:40.156 INFO:teuthology.orchestra.run.smithi167.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-19T01:19:40.355 INFO:teuthology.orchestra.run.smithi167.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-19T01:19:40.545 INFO:teuthology.orchestra.run.smithi167.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-19T01:19:40.780 INFO:teuthology.orchestra.run.smithi167.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-19T01:19:40.886 INFO:teuthology.orchestra.run.smithi167.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-19T01:19:41.192 INFO:teuthology.orchestra.run.smithi167.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-19T01:19:41.278 INFO:teuthology.orchestra.run.smithi167.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-19T01:19:41.396 INFO:teuthology.orchestra.run.smithi167.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-19T01:19:41.479 INFO:teuthology.orchestra.run.smithi167.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-19T01:19:41.567 INFO:teuthology.orchestra.run.smithi167.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-19T01:19:41.650 INFO:teuthology.orchestra.run.smithi167.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-19T01:19:41.749 INFO:teuthology.orchestra.run.smithi167.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-19T01:19:41.919 INFO:teuthology.orchestra.run.smithi167.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-19T01:19:42.063 INFO:teuthology.orchestra.run.smithi167.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2206-gb6ae2491-1jammy [5,085 kB] 2024-03-19T01:19:42.512 INFO:teuthology.orchestra.run.smithi167.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-19T01:19:42.596 INFO:teuthology.orchestra.run.smithi167.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-19T01:19:42.678 INFO:teuthology.orchestra.run.smithi167.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-19T01:19:42.777 INFO:teuthology.orchestra.run.smithi167.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-19T01:19:42.829 INFO:teuthology.orchestra.run.smithi167.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2206-gb6ae2491-1jammy [244 kB] 2024-03-19T01:19:42.863 INFO:teuthology.orchestra.run.smithi167.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-19T01:19:42.887 INFO:teuthology.orchestra.run.smithi167.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2206-gb6ae2491-1jammy [124 kB] 2024-03-19T01:19:42.899 INFO:teuthology.orchestra.run.smithi167.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2206-gb6ae2491-1jammy [1,526 kB] 2024-03-19T01:19:43.151 INFO:teuthology.orchestra.run.smithi167.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2206-gb6ae2491-1jammy [6,258 kB] 2024-03-19T01:19:44.231 INFO:teuthology.orchestra.run.smithi167.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2206-gb6ae2491-1jammy [22.8 MB] 2024-03-19T01:19:44.614 INFO:teuthology.orchestra.run.smithi007.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2206-gb6ae2491-1jammy [5,085 kB] 2024-03-19T01:19:45.565 INFO:teuthology.orchestra.run.smithi007.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2206-gb6ae2491-1jammy [244 kB] 2024-03-19T01:19:45.629 INFO:teuthology.orchestra.run.smithi007.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2206-gb6ae2491-1jammy [124 kB] 2024-03-19T01:19:45.636 INFO:teuthology.orchestra.run.smithi007.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2206-gb6ae2491-1jammy [1,526 kB] 2024-03-19T01:19:45.940 INFO:teuthology.orchestra.run.smithi007.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2206-gb6ae2491-1jammy [6,258 kB] 2024-03-19T01:19:47.386 INFO:teuthology.orchestra.run.smithi007.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2206-gb6ae2491-1jammy [22.8 MB] 2024-03-19T01:19:49.085 INFO:teuthology.orchestra.run.smithi167.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2206-gb6ae2491-1jammy [14.0 kB] 2024-03-19T01:19:49.086 INFO:teuthology.orchestra.run.smithi167.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2206-gb6ae2491-1jammy [870 kB] 2024-03-19T01:19:49.193 INFO:teuthology.orchestra.run.smithi167.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2206-gb6ae2491-1jammy [2,434 kB] 2024-03-19T01:19:49.706 INFO:teuthology.orchestra.run.smithi167.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2206-gb6ae2491-1jammy [777 kB] 2024-03-19T01:19:49.880 INFO:teuthology.orchestra.run.smithi167.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2206-gb6ae2491-1jammy [144 kB] 2024-03-19T01:19:49.890 INFO:teuthology.orchestra.run.smithi167.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2206-gb6ae2491-1jammy [1,936 kB] 2024-03-19T01:19:50.336 INFO:teuthology.orchestra.run.smithi167.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2206-gb6ae2491-1jammy [8,625 kB] 2024-03-19T01:19:52.222 INFO:teuthology.orchestra.run.smithi007.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2206-gb6ae2491-1jammy [14.0 kB] 2024-03-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi007.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2206-gb6ae2491-1jammy [870 kB] 2024-03-19T01:19:52.316 INFO:teuthology.orchestra.run.smithi007.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2206-gb6ae2491-1jammy [2,434 kB] 2024-03-19T01:19:52.369 INFO:teuthology.orchestra.run.smithi167.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2206-gb6ae2491-1jammy [14.2 kB] 2024-03-19T01:19:52.369 INFO:teuthology.orchestra.run.smithi167.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2206-gb6ae2491-1jammy [50.2 MB] 2024-03-19T01:19:52.808 INFO:teuthology.orchestra.run.smithi007.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2206-gb6ae2491-1jammy [777 kB] 2024-03-19T01:19:52.941 INFO:teuthology.orchestra.run.smithi007.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2206-gb6ae2491-1jammy [144 kB] 2024-03-19T01:19:52.957 INFO:teuthology.orchestra.run.smithi007.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2206-gb6ae2491-1jammy [1,936 kB] 2024-03-19T01:19:53.336 INFO:teuthology.orchestra.run.smithi007.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2206-gb6ae2491-1jammy [8,625 kB] 2024-03-19T01:19:55.043 INFO:teuthology.orchestra.run.smithi007.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2206-gb6ae2491-1jammy [14.2 kB] 2024-03-19T01:19:55.082 INFO:teuthology.orchestra.run.smithi007.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2206-gb6ae2491-1jammy [50.2 MB] 2024-03-19T01:20:00.131 INFO:teuthology.orchestra.run.smithi007.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2206-gb6ae2491-1jammy [121 kB] 2024-03-19T01:20:00.148 INFO:teuthology.orchestra.run.smithi167.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2206-gb6ae2491-1jammy [121 kB] 2024-03-19T01:20:00.151 INFO:teuthology.orchestra.run.smithi167.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2206-gb6ae2491-1jammy [229 kB] 2024-03-19T01:20:00.156 INFO:teuthology.orchestra.run.smithi167.stdout:Get:120 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2206-gb6ae2491-1jammy [34.0 kB] 2024-03-19T01:20:00.157 INFO:teuthology.orchestra.run.smithi167.stdout:Get:121 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2206-gb6ae2491-1jammy [23.4 kB] 2024-03-19T01:20:00.158 INFO:teuthology.orchestra.run.smithi167.stdout:Get:122 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2206-gb6ae2491-1jammy [38.9 kB] 2024-03-19T01:20:00.160 INFO:teuthology.orchestra.run.smithi167.stdout:Get:123 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2206-gb6ae2491-1jammy [13.2 MB] 2024-03-19T01:20:00.207 INFO:teuthology.orchestra.run.smithi007.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2206-gb6ae2491-1jammy [229 kB] 2024-03-19T01:20:00.212 INFO:teuthology.orchestra.run.smithi007.stdout:Get:120 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2206-gb6ae2491-1jammy [34.0 kB] 2024-03-19T01:20:00.213 INFO:teuthology.orchestra.run.smithi007.stdout:Get:121 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2206-gb6ae2491-1jammy [23.4 kB] 2024-03-19T01:20:00.214 INFO:teuthology.orchestra.run.smithi007.stdout:Get:122 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2206-gb6ae2491-1jammy [38.9 kB] 2024-03-19T01:20:00.215 INFO:teuthology.orchestra.run.smithi007.stdout:Get:123 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2206-gb6ae2491-1jammy [13.2 MB] 2024-03-19T01:20:01.093 INFO:teuthology.orchestra.run.smithi007.stdout:Get:124 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2206-gb6ae2491-1jammy [91.4 kB] 2024-03-19T01:20:01.269 INFO:teuthology.orchestra.run.smithi167.stdout:Get:124 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2206-gb6ae2491-1jammy [91.4 kB] 2024-03-19T01:20:01.729 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 178 MB in 28s (6,442 kB/s) 2024-03-19T01:20:01.884 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 178 MB in 28s (6,403 kB/s) 2024-03-19T01:20:01.914 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-19T01:20:02.015 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-19T01:20:02.020 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-19T01:20:02.021 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-19T01:20:02.062 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-19T01:20:02.107 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-19T01:20:02.112 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-19T01:20:02.143 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-19T01:20:02.374 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-19T01:20:02.402 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-19T01:20:02.440 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T01:20:02.455 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-19T01:20:02.481 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-19T01:20:02.521 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T01:20:02.744 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cmd2. 2024-03-19T01:20:02.770 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-19T01:20:02.801 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-19T01:20:02.858 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cmd2. 2024-03-19T01:20:02.884 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-19T01:20:02.924 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-19T01:20:03.105 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-19T01:20:03.131 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-19T01:20:03.197 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T01:20:03.277 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-19T01:20:03.304 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-19T01:20:03.386 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T01:20:03.525 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-19T01:20:03.541 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-19T01:20:03.583 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T01:20:03.781 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-19T01:20:03.808 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-19T01:20:03.848 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T01:20:03.920 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-19T01:20:03.946 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-19T01:20:03.986 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T01:20:04.143 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-19T01:20:04.169 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-19T01:20:04.217 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T01:20:04.424 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-19T01:20:04.450 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:20:04.490 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:20:04.680 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-19T01:20:04.706 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:20:04.754 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:20:04.827 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-19T01:20:04.854 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:20:04.893 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:20:05.158 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-19T01:20:05.184 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:20:05.224 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:20:05.238 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-19T01:20:05.265 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:20:05.305 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:20:05.527 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-19T01:20:05.554 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:20:05.594 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:20:05.650 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-19T01:20:05.677 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-19T01:20:05.716 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T01:20:05.964 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-19T01:20:05.990 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-19T01:20:06.558 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T01:20:06.728 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librados2. 2024-03-19T01:20:06.755 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../010-librados2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:06.794 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librados2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:07.226 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librados2. 2024-03-19T01:20:07.230 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librbd1. 2024-03-19T01:20:07.252 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../010-librados2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:07.254 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../011-librbd1_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:07.284 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librados2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:07.289 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librbd1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:07.667 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libcephfs2. 2024-03-19T01:20:07.694 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:07.725 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libcephfs2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:07.771 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librbd1. 2024-03-19T01:20:07.798 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../011-librbd1_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:07.837 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librbd1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:07.962 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rados. 2024-03-19T01:20:07.978 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:08.020 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rados (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:08.299 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libcephfs2. 2024-03-19T01:20:08.325 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:08.339 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-19T01:20:08.365 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libcephfs2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:08.366 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:08.408 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-ceph-argparse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:08.642 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rados. 2024-03-19T01:20:08.658 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cephfs. 2024-03-19T01:20:08.669 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:08.685 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:08.709 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rados (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:08.725 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cephfs (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:08.995 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-19T01:20:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-19T01:20:09.021 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:09.026 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:09.061 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:09.061 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-ceph-argparse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:09.314 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cephfs. 2024-03-19T01:20:09.322 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-prettytable. 2024-03-19T01:20:09.330 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:09.338 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-19T01:20:09.372 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-19T01:20:09.372 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cephfs (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:09.625 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rbd. 2024-03-19T01:20:09.652 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:09.675 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-19T01:20:09.691 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:09.702 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:09.742 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:10.037 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-19T01:20:10.046 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-prettytable. 2024-03-19T01:20:10.063 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-19T01:20:10.072 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-19T01:20:10.103 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T01:20:10.103 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-19T01:20:10.366 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rbd. 2024-03-19T01:20:10.382 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:10.389 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-19T01:20:10.405 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-19T01:20:10.425 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:10.447 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T01:20:10.725 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-19T01:20:10.741 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-19T01:20:10.778 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-19T01:20:10.783 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T01:20:10.804 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-19T01:20:10.844 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T01:20:11.112 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua5.1. 2024-03-19T01:20:11.113 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-19T01:20:11.128 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-19T01:20:11.139 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-19T01:20:11.163 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T01:20:11.178 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-19T01:20:11.475 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-19T01:20:11.490 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-19T01:20:11.523 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-any. 2024-03-19T01:20:11.533 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T01:20:11.550 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-19T01:20:11.590 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-19T01:20:11.853 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package lua5.1. 2024-03-19T01:20:11.879 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-19T01:20:11.928 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-19T01:20:12.001 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package zip. 2024-03-19T01:20:12.028 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-19T01:20:12.059 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking zip (3.0-12build2) ... 2024-03-19T01:20:12.239 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package lua-any. 2024-03-19T01:20:12.265 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-19T01:20:12.306 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-19T01:20:12.380 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package unzip. 2024-03-19T01:20:12.406 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-19T01:20:12.446 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-19T01:20:12.533 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package zip. 2024-03-19T01:20:12.560 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-19T01:20:12.600 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking zip (3.0-12build2) ... 2024-03-19T01:20:12.850 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package luarocks. 2024-03-19T01:20:12.877 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-19T01:20:12.917 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-19T01:20:12.920 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package unzip. 2024-03-19T01:20:12.947 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-19T01:20:12.986 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-19T01:20:13.258 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librgw2. 2024-03-19T01:20:13.285 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../027-librgw2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:13.316 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librgw2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:13.399 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package luarocks. 2024-03-19T01:20:13.426 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-19T01:20:13.466 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-19T01:20:13.815 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librgw2. 2024-03-19T01:20:13.830 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../027-librgw2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:13.861 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rgw. 2024-03-19T01:20:13.864 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librgw2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:13.876 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:13.919 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:14.255 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-19T01:20:14.282 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-19T01:20:14.322 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T01:20:14.392 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rgw. 2024-03-19T01:20:14.408 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:14.450 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:14.592 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libradosstriper1. 2024-03-19T01:20:14.612 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:14.650 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libradosstriper1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:14.795 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-19T01:20:14.822 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-19T01:20:14.861 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T01:20:15.012 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-common. 2024-03-19T01:20:15.038 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:15.078 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:15.148 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libradosstriper1. 2024-03-19T01:20:15.175 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:15.231 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libradosstriper1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:15.643 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-common. 2024-03-19T01:20:15.670 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:15.709 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:16.459 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-base. 2024-03-19T01:20:16.486 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:16.535 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-base (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:17.080 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-19T01:20:17.107 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-19T01:20:17.146 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-19T01:20:17.315 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-base. 2024-03-19T01:20:17.341 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:17.391 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-base (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:17.407 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cheroot. 2024-03-19T01:20:17.424 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-19T01:20:17.457 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T01:20:17.711 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-19T01:20:17.738 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-19T01:20:17.769 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-19T01:20:17.952 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-19T01:20:17.979 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-19T01:20:18.019 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-19T01:20:18.072 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-19T01:20:18.099 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-19T01:20:18.138 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-19T01:20:18.422 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cheroot. 2024-03-19T01:20:18.441 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-19T01:20:18.448 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-19T01:20:18.469 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-19T01:20:18.496 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T01:20:18.508 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-19T01:20:18.769 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-tempora. 2024-03-19T01:20:18.785 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-19T01:20:18.819 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-19T01:20:18.866 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-19T01:20:18.893 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-19T01:20:18.941 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-19T01:20:19.114 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-portend. 2024-03-19T01:20:19.141 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-19T01:20:19.180 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-19T01:20:19.336 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-19T01:20:19.363 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-19T01:20:19.403 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-19T01:20:19.484 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-19T01:20:19.511 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-19T01:20:19.550 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-19T01:20:19.705 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-19T01:20:19.732 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-19T01:20:19.772 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-19T01:20:19.828 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-19T01:20:19.855 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-19T01:20:19.886 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-19T01:20:20.117 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-tempora. 2024-03-19T01:20:20.144 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-19T01:20:20.208 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-19T01:20:20.232 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-natsort. 2024-03-19T01:20:20.259 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-19T01:20:20.298 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-19T01:20:20.578 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-portend. 2024-03-19T01:20:20.604 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-19T01:20:20.610 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-logutils. 2024-03-19T01:20:20.637 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-19T01:20:20.653 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-19T01:20:20.676 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-19T01:20:20.938 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-19T01:20:20.954 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-19T01:20:20.972 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-19T01:20:20.987 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-19T01:20:20.999 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-19T01:20:21.038 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-19T01:20:21.291 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-mako. 2024-03-19T01:20:21.317 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-19T01:20:21.350 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-19T01:20:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T01:20:21.377 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-19T01:20:21.416 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-19T01:20:21.694 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-19T01:20:21.721 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-19T01:20:21.760 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-19T01:20:21.770 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-natsort. 2024-03-19T01:20:21.797 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-19T01:20:21.828 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-19T01:20:22.063 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-19T01:20:22.090 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-19T01:20:22.106 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-logutils. 2024-03-19T01:20:22.130 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-19T01:20:22.133 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-19T01:20:22.173 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-19T01:20:22.485 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-19T01:20:22.511 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-19T01:20:22.551 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-19T01:20:22.594 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-webob. 2024-03-19T01:20:22.621 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-19T01:20:22.660 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-19T01:20:22.787 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-mako. 2024-03-19T01:20:22.814 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-19T01:20:22.854 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T01:20:23.005 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-waitress. 2024-03-19T01:20:23.032 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-19T01:20:23.074 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T01:20:23.198 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-19T01:20:23.225 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-19T01:20:23.265 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-19T01:20:23.399 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-tempita. 2024-03-19T01:20:23.426 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-19T01:20:23.465 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T01:20:23.543 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-19T01:20:23.570 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-19T01:20:23.610 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-19T01:20:23.760 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-paste. 2024-03-19T01:20:23.788 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-19T01:20:23.827 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T01:20:23.887 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-webob. 2024-03-19T01:20:23.914 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-19T01:20:23.954 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-19T01:20:24.154 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-19T01:20:24.171 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-19T01:20:24.213 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T01:20:24.290 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-waitress. 2024-03-19T01:20:24.317 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-19T01:20:24.361 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T01:20:24.516 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-19T01:20:24.543 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-19T01:20:24.582 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-19T01:20:24.652 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-tempita. 2024-03-19T01:20:24.678 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-19T01:20:24.718 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T01:20:24.835 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-webtest. 2024-03-19T01:20:24.862 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-19T01:20:24.893 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-19T01:20:25.004 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-paste. 2024-03-19T01:20:25.031 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-19T01:20:25.071 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T01:20:25.196 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pecan. 2024-03-19T01:20:25.223 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-19T01:20:25.263 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T01:20:25.399 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-19T01:20:25.414 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-19T01:20:25.457 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T01:20:25.601 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-certifi. 2024-03-19T01:20:25.628 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-19T01:20:25.667 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-19T01:20:25.777 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-19T01:20:25.803 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-19T01:20:25.843 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-19T01:20:25.987 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-idna. 2024-03-19T01:20:26.014 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-19T01:20:26.053 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-19T01:20:26.121 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-webtest. 2024-03-19T01:20:26.148 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-19T01:20:26.188 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-19T01:20:26.373 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-urllib3. 2024-03-19T01:20:26.400 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-19T01:20:26.439 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T01:20:26.449 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pecan. 2024-03-19T01:20:26.476 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-19T01:20:26.515 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T01:20:26.768 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-requests. 2024-03-19T01:20:26.795 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-19T01:20:26.834 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T01:20:26.879 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-certifi. 2024-03-19T01:20:26.906 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-19T01:20:26.945 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-19T01:20:27.120 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-19T01:20:27.137 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-19T01:20:27.178 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T01:20:27.265 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-idna. 2024-03-19T01:20:27.291 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-19T01:20:27.331 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-19T01:20:27.623 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-19T01:20:27.651 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:27.706 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:28.001 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-urllib3. 2024-03-19T01:20:28.028 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-19T01:20:28.068 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T01:20:28.103 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-19T01:20:28.130 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:28.169 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:28.412 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-requests. 2024-03-19T01:20:28.439 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-19T01:20:28.479 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T01:20:28.490 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-19T01:20:28.516 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-19T01:20:28.555 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-19T01:20:28.772 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-19T01:20:28.788 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-19T01:20:28.825 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T01:20:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-openssl. 2024-03-19T01:20:28.902 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-19T01:20:28.941 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-19T01:20:29.110 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-19T01:20:29.125 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:29.168 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:29.227 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr. 2024-03-19T01:20:29.244 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:29.277 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:29.556 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-19T01:20:29.583 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:29.623 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:29.647 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mon. 2024-03-19T01:20:29.674 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:29.714 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mon (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:29.951 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-19T01:20:29.977 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-19T01:20:30.017 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-19T01:20:30.304 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-openssl. 2024-03-19T01:20:30.326 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-osd. 2024-03-19T01:20:30.330 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-19T01:20:30.353 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:30.370 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-19T01:20:30.392 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-osd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:30.706 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr. 2024-03-19T01:20:30.733 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:30.773 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:31.168 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mon. 2024-03-19T01:20:31.195 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:31.234 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mon (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:31.386 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph. 2024-03-19T01:20:31.400 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../068-ceph_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:31.428 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:31.681 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-fuse. 2024-03-19T01:20:31.697 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:31.739 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:31.788 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-osd. 2024-03-19T01:20:31.814 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:31.846 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-osd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:32.109 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mds. 2024-03-19T01:20:32.136 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:32.175 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mds (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:32.595 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephadm. 2024-03-19T01:20:32.623 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../071-cephadm_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:32.661 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:33.033 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph. 2024-03-19T01:20:33.059 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../068-ceph_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:33.099 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-19T01:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-19T01:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-19T01:20:33.418 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-fuse. 2024-03-19T01:20:33.445 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:33.485 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:33.500 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-19T01:20:33.527 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T01:20:33.566 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T01:20:33.871 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mds. 2024-03-19T01:20:33.898 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:33.937 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mds (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:34.291 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package cephadm. 2024-03-19T01:20:34.306 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../071-cephadm_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:34.340 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:34.605 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-babel. 2024-03-19T01:20:34.632 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T01:20:34.671 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T01:20:34.685 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-19T01:20:34.712 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-19T01:20:34.751 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-19T01:20:35.007 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jinja2. 2024-03-19T01:20:35.035 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-19T01:20:35.074 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T01:20:35.079 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-19T01:20:35.106 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T01:20:35.146 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T01:20:35.436 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-19T01:20:35.463 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:35.502 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:35.822 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-19T01:20:35.838 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-19T01:20:35.880 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-19T01:20:36.159 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-babel. 2024-03-19T01:20:36.174 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-routes. 2024-03-19T01:20:36.187 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T01:20:36.202 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-19T01:20:36.226 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T01:20:36.240 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T01:20:36.554 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jinja2. 2024-03-19T01:20:36.581 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-19T01:20:36.585 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-19T01:20:36.613 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:36.620 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T01:20:36.652 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:36.982 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-19T01:20:37.009 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:37.048 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:37.105 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-19T01:20:37.163 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T01:20:37.427 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-19T01:20:37.453 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-19T01:20:37.493 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-19T01:20:37.683 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-joblib. 2024-03-19T01:20:37.700 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-19T01:20:37.729 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-routes. 2024-03-19T01:20:37.741 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T01:20:37.744 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-19T01:20:37.779 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T01:20:38.070 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-19T01:20:38.082 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-19T01:20:38.086 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-19T01:20:38.109 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:38.140 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:38.214 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-19T01:20:38.508 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-sklearn. 2024-03-19T01:20:38.536 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-19T01:20:38.574 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T01:20:38.594 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-19T01:20:38.621 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-19T01:20:38.660 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T01:20:39.147 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-joblib. 2024-03-19T01:20:39.174 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-19T01:20:39.213 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T01:20:39.242 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-19T01:20:39.270 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:39.308 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:39.559 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-19T01:20:39.586 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-19T01:20:39.719 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-19T01:20:40.030 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-sklearn. 2024-03-19T01:20:40.058 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-19T01:20:40.097 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T01:20:40.270 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cachetools. 2024-03-19T01:20:40.298 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-19T01:20:40.336 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-19T01:20:40.598 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-19T01:20:40.625 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-19T01:20:40.656 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-19T01:20:40.839 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-19T01:20:40.867 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:40.906 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:40.959 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-19T01:20:40.987 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-19T01:20:41.025 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T01:20:41.312 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rsa. 2024-03-19T01:20:41.340 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-19T01:20:41.378 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-19T01:20:41.699 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-google-auth. 2024-03-19T01:20:41.726 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-19T01:20:41.782 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-19T01:20:41.867 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cachetools. 2024-03-19T01:20:41.895 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-19T01:20:41.934 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-19T01:20:42.102 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-19T01:20:42.130 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-19T01:20:42.168 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T01:20:42.237 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-19T01:20:42.253 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-19T01:20:42.287 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-19T01:20:42.446 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-websocket. 2024-03-19T01:20:42.474 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-19T01:20:42.504 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-19T01:20:42.607 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-19T01:20:42.634 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-19T01:20:42.673 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T01:20:42.808 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-19T01:20:42.835 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-19T01:20:42.874 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T01:20:42.985 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rsa. 2024-03-19T01:20:43.012 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-19T01:20:43.051 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-19T01:20:43.384 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-google-auth. 2024-03-19T01:20:43.412 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-19T01:20:43.590 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-19T01:20:43.605 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-19T01:20:43.633 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:43.671 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:43.900 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-19T01:20:43.927 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-19T01:20:43.957 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-19T01:20:43.957 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T01:20:43.985 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-19T01:20:44.024 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T01:20:44.255 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-websocket. 2024-03-19T01:20:44.280 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-19T01:20:44.323 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-19T01:20:44.335 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-19T01:20:44.372 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T01:20:44.401 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T01:20:44.640 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-19T01:20:44.665 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package jq. 2024-03-19T01:20:44.669 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-19T01:20:44.679 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T01:20:44.707 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T01:20:44.721 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-19T01:20:44.974 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package socat. 2024-03-19T01:20:44.990 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-19T01:20:45.031 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-19T01:20:45.343 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package xmlstarlet. 2024-03-19T01:20:45.362 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-19T01:20:45.371 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-19T01:20:45.390 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:45.409 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-19T01:20:45.428 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:45.689 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-19T01:20:45.716 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-19T01:20:45.747 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T01:20:45.780 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-test. 2024-03-19T01:20:45.797 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:45.838 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-test (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:46.125 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-19T01:20:46.153 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T01:20:46.191 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T01:20:46.486 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package jq. 2024-03-19T01:20:46.514 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T01:20:46.553 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-19T01:20:46.788 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package socat. 2024-03-19T01:20:46.805 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-19T01:20:46.838 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-19T01:20:47.091 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package xmlstarlet. 2024-03-19T01:20:47.108 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-19T01:20:47.141 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-19T01:20:47.488 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-test. 2024-03-19T01:20:47.515 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:47.554 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-test (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-volume. 2024-03-19T01:20:48.471 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:48.500 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-volume (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:48.838 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-19T01:20:48.866 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:48.905 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephfs-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:49.158 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-colorama. 2024-03-19T01:20:49.186 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-19T01:20:49.215 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-19T01:20:49.451 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephfs-shell. 2024-03-19T01:20:49.468 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:49.509 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephfs-shell (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:49.771 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephfs-top. 2024-03-19T01:20:49.795 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:49.837 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephfs-top (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:49.858 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-volume. 2024-03-19T01:20:49.886 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:49.924 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-volume (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:50.090 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-19T01:20:50.106 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:50.148 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libcephfs-dev (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:50.231 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-19T01:20:50.259 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:50.289 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking cephfs-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:50.475 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-19T01:20:50.503 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-19T01:20:50.542 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T01:20:50.609 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-colorama. 2024-03-19T01:20:50.637 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-19T01:20:50.675 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-19T01:20:50.846 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-19T01:20:50.874 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-19T01:20:50.904 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-19T01:20:50.936 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package cephfs-shell. 2024-03-19T01:20:50.963 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:51.011 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking cephfs-shell (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:51.250 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package nvme-cli. 2024-03-19T01:20:51.278 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-19T01:20:51.280 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package cephfs-top. 2024-03-19T01:20:51.308 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:20:51.316 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T01:20:51.347 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking cephfs-top (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:51.566 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-19T01:20:51.582 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:51.624 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libcephfs-dev (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:51.722 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package pkg-config. 2024-03-19T01:20:51.750 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-19T01:20:51.779 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T01:20:51.944 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-19T01:20:51.972 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-19T01:20:52.011 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T01:20:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-19T01:20:52.057 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-19T01:20:52.099 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-19T01:20:52.340 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-19T01:20:52.368 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-19T01:20:52.407 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-19T01:20:52.602 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-19T01:20:52.630 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-19T01:20:52.668 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-19T01:20:52.753 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package nvme-cli. 2024-03-19T01:20:52.781 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-19T01:20:52.819 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T01:20:52.971 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pastescript. 2024-03-19T01:20:52.999 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-19T01:20:53.063 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-19T01:20:53.200 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package pkg-config. 2024-03-19T01:20:53.227 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-19T01:20:53.258 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T01:20:53.391 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pluggy. 2024-03-19T01:20:53.419 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-19T01:20:53.458 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-19T01:20:53.519 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-19T01:20:53.535 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-19T01:20:53.577 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-19T01:20:53.777 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-psutil. 2024-03-19T01:20:53.805 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-19T01:20:53.844 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-19T01:20:54.022 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-19T01:20:54.050 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-19T01:20:54.089 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-19T01:20:54.205 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-py. 2024-03-19T01:20:54.233 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-19T01:20:54.280 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-19T01:20:54.392 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pastescript. 2024-03-19T01:20:54.419 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-19T01:20:54.458 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-19T01:20:54.600 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pygments. 2024-03-19T01:20:54.628 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-19T01:20:54.666 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T01:20:54.778 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pluggy. 2024-03-19T01:20:54.806 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-19T01:20:54.853 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-19T01:20:55.137 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-19T01:20:55.140 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-psutil. 2024-03-19T01:20:55.166 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-19T01:20:55.168 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-19T01:20:55.195 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-19T01:20:55.198 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-19T01:20:55.481 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-toml. 2024-03-19T01:20:55.510 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-19T01:20:55.543 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-py. 2024-03-19T01:20:55.547 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-19T01:20:55.571 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-19T01:20:55.617 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-19T01:20:55.850 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pytest. 2024-03-19T01:20:55.882 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-19T01:20:55.916 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T01:20:55.921 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pygments. 2024-03-19T01:20:55.949 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-19T01:20:55.988 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T01:20:56.279 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-simplejson. 2024-03-19T01:20:56.307 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-19T01:20:56.345 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-19T01:20:56.401 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-19T01:20:56.416 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-19T01:20:56.450 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-19T01:20:56.598 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-19T01:20:56.615 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-19T01:20:56.656 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-19T01:20:56.744 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-toml. 2024-03-19T01:20:56.772 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-19T01:20:56.811 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-19T01:20:57.114 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pytest. 2024-03-19T01:20:57.142 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-19T01:20:57.180 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T01:20:57.318 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package radosgw. 2024-03-19T01:20:57.335 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../122-radosgw_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:57.376 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking radosgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:57.551 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-simplejson. 2024-03-19T01:20:57.579 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-19T01:20:57.617 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-19T01:20:57.870 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-19T01:20:57.886 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-19T01:20:57.928 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-19T01:20:58.163 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package rbd-fuse. 2024-03-19T01:20:58.179 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:58.212 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking rbd-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:58.534 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-19T01:20:58.776 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T01:20:58.892 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package radosgw. 2024-03-19T01:20:58.901 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T01:20:58.919 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../122-radosgw_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:59.040 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking radosgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:59.154 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-19T01:20:59.489 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-19T01:20:59.860 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package rbd-fuse. 2024-03-19T01:20:59.888 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:20:59.916 INFO:teuthology.orchestra.run.smithi007.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-19T01:20:59.929 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking rbd-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:20:59.982 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:00.240 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-19T01:21:00.437 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user cephadm....done 2024-03-19T01:21:00.490 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T01:21:00.576 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T01:21:00.617 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T01:21:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-19T01:21:00.884 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-19T01:21:01.120 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-19T01:21:01.220 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-19T01:21:01.238 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-19T01:21:01.505 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-19T01:21:01.664 INFO:teuthology.orchestra.run.smithi167.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-19T01:21:01.727 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:01.773 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T01:21:01.899 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-19T01:21:02.209 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T01:21:02.227 INFO:teuthology.orchestra.run.smithi167.stdout:Adding system user cephadm....done 2024-03-19T01:21:02.383 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T01:21:02.572 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-19T01:21:02.660 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-19T01:21:02.842 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-19T01:21:02.910 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-19T01:21:03.029 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-19T01:21:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-19T01:21:03.280 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-19T01:21:03.376 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-19T01:21:03.539 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T01:21:03.644 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-19T01:21:03.665 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-19T01:21:03.967 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T01:21:04.020 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-19T01:21:04.325 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-19T01:21:04.327 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-19T01:21:04.579 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-19T01:21:04.680 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-ceph-argparse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:04.822 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-19T01:21:04.948 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T01:21:05.074 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T01:21:05.126 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-19T01:21:05.200 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-19T01:21:05.341 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-19T01:21:05.460 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T01:21:05.584 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-19T01:21:05.585 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-19T01:21:05.844 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-19T01:21:05.862 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-19T01:21:06.163 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-ceph-argparse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:06.220 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-19T01:21:06.251 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-19T01:21:06.315 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T01:21:06.441 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-19T01:21:06.513 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T01:21:06.639 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T01:21:06.756 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-19T01:21:06.826 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T01:21:06.944 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-19T01:21:06.999 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T01:21:07.125 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-19T01:21:07.203 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T01:21:07.376 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-19T01:21:07.479 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-19T01:21:07.759 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-19T01:21:07.781 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up zip (3.0-12build2) ... 2024-03-19T01:21:07.790 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-19T01:21:07.847 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T01:21:07.923 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T01:21:07.964 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-19T01:21:08.324 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T01:21:08.450 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-19T01:21:08.625 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T01:21:08.726 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T01:21:08.912 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T01:21:09.011 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-19T01:21:09.019 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-19T01:21:09.137 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T01:21:09.337 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up zip (3.0-12build2) ... 2024-03-19T01:21:09.438 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T01:21:09.446 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-19T01:21:09.839 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-19T01:21:10.057 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T01:21:10.074 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T01:21:10.291 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T01:21:10.417 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-19T01:21:10.450 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T01:21:10.543 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T01:21:10.752 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-19T01:21:10.852 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-19T01:21:11.061 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T01:21:11.262 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-19T01:21:11.413 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-19T01:21:11.514 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T01:21:11.664 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-19T01:21:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:11.865 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T01:21:12.091 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-idna (3.3-1) ... 2024-03-19T01:21:12.133 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-19T01:21:12.409 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T01:21:12.426 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T01:21:12.769 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-19T01:21:13.037 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-19T01:21:13.163 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:13.448 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-idna (3.3-1) ... 2024-03-19T01:21:13.623 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T01:21:13.749 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T01:21:13.858 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:21:13.993 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-19T01:21:14.260 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T01:21:14.386 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-19T01:21:14.513 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T01:21:14.814 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-19T01:21:14.830 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T01:21:15.057 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-19T01:21:15.082 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:21:15.216 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-19T01:21:15.300 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:21:15.426 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-19T01:21:15.476 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T01:21:15.602 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-19T01:21:15.727 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-19T01:21:15.728 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T01:21:15.978 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-19T01:21:16.013 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-19T01:21:16.246 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-19T01:21:16.255 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-19T01:21:16.490 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:21:16.506 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-19T01:21:16.599 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-19T01:21:16.774 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-19T01:21:16.909 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-19T01:21:17.033 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T01:21:17.159 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-19T01:21:17.177 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-19T01:21:17.427 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T01:21:17.436 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-19T01:21:17.553 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T01:21:17.704 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-19T01:21:17.846 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-19T01:21:17.939 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-19T01:21:18.147 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-19T01:21:18.206 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T01:21:18.332 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-19T01:21:18.406 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T01:21:18.532 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-19T01:21:18.625 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T01:21:18.658 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T01:21:18.747 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-19T01:21:18.751 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T01:21:18.985 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T01:21:19.044 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-19T01:21:19.145 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T01:21:19.346 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-19T01:21:19.538 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-19T01:21:19.714 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T01:21:19.806 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-19T01:21:19.823 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-19T01:21:19.907 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-19T01:21:19.924 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T01:21:20.013 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-19T01:21:20.150 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:21:20.251 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T01:21:20.275 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T01:21:20.377 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T01:21:20.593 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T01:21:20.770 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-19T01:21:20.878 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T01:21:21.030 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-19T01:21:21.156 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-19T01:21:21.179 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-19T01:21:21.407 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:21:21.407 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-19T01:21:21.533 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T01:21:21.667 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-19T01:21:21.850 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T01:21:21.918 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T01:21:22.151 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T01:21:22.170 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-19T01:21:22.420 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-19T01:21:22.589 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-19T01:21:22.631 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-19T01:21:22.890 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T01:21:22.891 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-19T01:21:23.150 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T01:21:23.226 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T01:21:23.327 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librados2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:23.410 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-19T01:21:23.436 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:23.563 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-19T01:21:23.779 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-19T01:21:23.866 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libcephfs2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:23.975 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libradosstriper1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:24.056 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T01:21:24.101 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-19T01:21:24.383 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T01:21:24.478 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librbd1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:24.509 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librados2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:24.604 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-modules-core (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:24.635 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:24.730 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:24.761 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-19T01:21:24.948 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T01:21:24.948 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T01:21:25.057 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libcephfs2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:25.182 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libradosstriper1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:25.308 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-19T01:21:25.440 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libcephfs-dev (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:25.566 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rados (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:25.585 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librbd1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:25.692 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librgw2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:25.702 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-modules-core (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:25.793 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephfs-top (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:25.837 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:25.893 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T01:21:26.048 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T01:21:26.048 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T01:21:26.538 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libcephfs-dev (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:26.665 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rados (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:26.791 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librgw2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:26.916 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up cephfs-top (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:27.042 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T01:21:27.181 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:27.282 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up rbd-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:27.383 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:27.509 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cephfs (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:27.635 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephfs-shell (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:27.761 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:28.238 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:28.304 INFO:teuthology.orchestra.run.smithi007.stdout:Adding group ceph....done 2024-03-19T01:21:28.364 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up rbd-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:28.490 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:28.616 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cephfs (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:28.656 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user ceph....done 2024-03-19T01:21:28.742 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up cephfs-shell (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:28.824 INFO:teuthology.orchestra.run.smithi007.stdout:Setting system user ceph properties....done 2024-03-19T01:21:28.838 INFO:teuthology.orchestra.run.smithi007.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-19T01:21:28.868 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:28.981 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-19T01:21:29.328 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-19T01:21:29.370 INFO:teuthology.orchestra.run.smithi167.stdout:Adding group ceph....done 2024-03-19T01:21:29.697 INFO:teuthology.orchestra.run.smithi167.stdout:Adding system user ceph....done 2024-03-19T01:21:29.829 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephfs-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:29.857 INFO:teuthology.orchestra.run.smithi167.stdout:Setting system user ceph properties....done 2024-03-19T01:21:29.872 INFO:teuthology.orchestra.run.smithi167.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-19T01:21:30.011 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-19T01:21:30.053 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:21:30.053 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:21:30.352 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-19T01:21:30.564 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-test (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:30.724 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up radosgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:30.836 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up cephfs-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:31.046 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:21:31.046 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:21:31.171 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:21:31.171 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:21:31.530 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-test (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:31.651 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-base (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:31.655 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up radosgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:32.009 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-19T01:21:32.162 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:21:32.162 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:21:32.495 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mds (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:32.649 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-base (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:32.750 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T01:21:32.751 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T01:21:32.998 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-19T01:21:33.222 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:33.446 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T01:21:33.446 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T01:21:33.509 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mds (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:33.724 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T01:21:33.725 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T01:21:33.923 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-osd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:34.195 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:34.231 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T01:21:34.231 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T01:21:34.412 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T01:21:34.412 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T01:21:35.013 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:35.041 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-osd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:35.139 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:35.290 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mon (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:35.335 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T01:21:35.335 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T01:21:35.511 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T01:21:35.511 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T01:21:35.854 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:36.003 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:36.016 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:36.142 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:36.146 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mon (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:36.243 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-dashboard (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:36.355 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T01:21:36.355 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T01:21:36.377 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-volume (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:36.739 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-19T01:21:36.857 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T01:21:36.875 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:37.009 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T01:21:37.031 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:37.157 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-dashboard (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:37.309 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-volume (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:21:37.645 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-19T01:21:37.737 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T01:21:37.884 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T01:21:38.052 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T01:21:38.756 INFO:teuthology.orchestra.run.smithi007.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T01:21:38.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-19T01:21:38.820 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-03-19T01:21:38.943 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T01:21:39.029 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-03-19T01:21:39.030 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-03-19T01:21:39.220 INFO:teuthology.orchestra.run.smithi007.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-19T01:21:39.221 INFO:teuthology.orchestra.run.smithi007.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-19T01:21:39.221 INFO:teuthology.orchestra.run.smithi007.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-19T01:21:39.221 INFO:teuthology.orchestra.run.smithi007.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-19T01:21:39.221 INFO:teuthology.orchestra.run.smithi007.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-19T01:21:39.221 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:21:39.221 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:21:39.221 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:21:39.221 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:21:39.222 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:21:39.222 INFO:teuthology.orchestra.run.smithi007.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:21:39.222 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:21:39.223 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2024-03-19T01:21:39.223 INFO:teuthology.orchestra.run.smithi007.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-19T01:21:39.223 INFO:teuthology.orchestra.run.smithi007.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-19T01:21:39.223 INFO:teuthology.orchestra.run.smithi007.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-19T01:21:39.223 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-19T01:21:39.223 INFO:teuthology.orchestra.run.smithi007.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-19T01:21:39.225 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2024-03-19T01:21:39.225 INFO:teuthology.orchestra.run.smithi007.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-19T01:21:39.225 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-19T01:21:39.225 INFO:teuthology.orchestra.run.smithi007.stdout: libengine-pkcs11-openssl 2024-03-19T01:21:39.266 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2024-03-19T01:21:39.266 INFO:teuthology.orchestra.run.smithi007.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-19T01:21:39.267 INFO:teuthology.orchestra.run.smithi007.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-19T01:21:39.267 INFO:teuthology.orchestra.run.smithi007.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-19T01:21:39.267 INFO:teuthology.orchestra.run.smithi007.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-19T01:21:39.267 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-19T01:21:39.268 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-19T01:21:39.268 INFO:teuthology.orchestra.run.smithi007.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-19T01:21:39.269 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages will be upgraded: 2024-03-19T01:21:39.270 INFO:teuthology.orchestra.run.smithi007.stdout: libssl3 2024-03-19T01:21:39.329 INFO:teuthology.orchestra.run.smithi007.stdout:1 upgraded, 39 newly installed, 0 to remove and 284 not upgraded. 2024-03-19T01:21:39.329 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 54.6 MB of archives. 2024-03-19T01:21:39.329 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-19T01:21:39.330 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-19T01:21:39.445 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-19T01:21:39.509 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-19T01:21:39.518 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-19T01:21:39.519 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-19T01:21:39.521 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-19T01:21:39.522 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-19T01:21:39.523 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-19T01:21:39.524 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-19T01:21:39.525 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-19T01:21:39.525 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-19T01:21:39.531 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-19T01:21:39.546 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-19T01:21:39.567 INFO:teuthology.orchestra.run.smithi167.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T01:21:39.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-19T01:21:39.622 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-03-19T01:21:39.818 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2024-03-19T01:21:39.819 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2024-03-19T01:21:39.880 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-19T01:21:39.881 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-19T01:21:39.882 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-19T01:21:39.884 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-19T01:21:39.884 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-19T01:21:39.889 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-19T01:21:39.890 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-19T01:21:39.897 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-19T01:21:39.898 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-19T01:21:39.919 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-19T01:21:39.920 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-19T01:21:39.930 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-19T01:21:39.954 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-19T01:21:39.957 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-19T01:21:39.958 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-19T01:21:39.958 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-19T01:21:39.959 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-19T01:21:39.970 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-19T01:21:39.971 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-19T01:21:39.984 INFO:teuthology.orchestra.run.smithi167.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-19T01:21:39.984 INFO:teuthology.orchestra.run.smithi167.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-19T01:21:39.984 INFO:teuthology.orchestra.run.smithi167.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-19T01:21:39.984 INFO:teuthology.orchestra.run.smithi167.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-19T01:21:39.984 INFO:teuthology.orchestra.run.smithi167.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-19T01:21:39.984 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:21:39.984 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:21:39.985 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:21:39.985 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:21:39.985 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:21:39.985 INFO:teuthology.orchestra.run.smithi167.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:21:39.985 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:21:39.986 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2024-03-19T01:21:39.986 INFO:teuthology.orchestra.run.smithi167.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-19T01:21:39.986 INFO:teuthology.orchestra.run.smithi167.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-19T01:21:39.986 INFO:teuthology.orchestra.run.smithi167.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-19T01:21:39.986 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-19T01:21:39.986 INFO:teuthology.orchestra.run.smithi167.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-19T01:21:39.987 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2024-03-19T01:21:39.987 INFO:teuthology.orchestra.run.smithi167.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-19T01:21:39.987 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-19T01:21:39.988 INFO:teuthology.orchestra.run.smithi167.stdout: libengine-pkcs11-openssl 2024-03-19T01:21:40.025 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2024-03-19T01:21:40.026 INFO:teuthology.orchestra.run.smithi167.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-19T01:21:40.026 INFO:teuthology.orchestra.run.smithi167.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-19T01:21:40.026 INFO:teuthology.orchestra.run.smithi167.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-19T01:21:40.026 INFO:teuthology.orchestra.run.smithi167.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-19T01:21:40.026 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-19T01:21:40.026 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-19T01:21:40.027 INFO:teuthology.orchestra.run.smithi167.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-19T01:21:40.027 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages will be upgraded: 2024-03-19T01:21:40.028 INFO:teuthology.orchestra.run.smithi167.stdout: libssl3 2024-03-19T01:21:40.096 INFO:teuthology.orchestra.run.smithi167.stdout:1 upgraded, 39 newly installed, 0 to remove and 284 not upgraded. 2024-03-19T01:21:40.096 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 54.6 MB of archives. 2024-03-19T01:21:40.096 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-19T01:21:40.096 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-19T01:21:40.188 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-19T01:21:40.188 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-19T01:21:40.188 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-19T01:21:40.188 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-19T01:21:40.188 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-19T01:21:40.188 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-19T01:21:40.200 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-19T01:21:40.200 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-19T01:21:40.225 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-19T01:21:40.292 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-19T01:21:40.307 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-19T01:21:40.308 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-19T01:21:40.311 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-19T01:21:40.313 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-19T01:21:40.313 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-19T01:21:40.315 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-19T01:21:40.316 INFO:teuthology.orchestra.run.smithi167.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-19T01:21:40.317 INFO:teuthology.orchestra.run.smithi167.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-19T01:21:40.318 INFO:teuthology.orchestra.run.smithi167.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-19T01:21:40.337 INFO:teuthology.orchestra.run.smithi167.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-19T01:21:40.618 INFO:teuthology.orchestra.run.smithi007.stdout:Preconfiguring packages ... 2024-03-19T01:21:40.813 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 54.6 MB in 1s (59.2 MB/s) 2024-03-19T01:21:40.859 INFO:teuthology.orchestra.run.smithi167.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-19T01:21:40.862 INFO:teuthology.orchestra.run.smithi167.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-19T01:21:40.862 INFO:teuthology.orchestra.run.smithi167.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-19T01:21:40.866 INFO:teuthology.orchestra.run.smithi167.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-19T01:21:40.867 INFO:teuthology.orchestra.run.smithi167.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-19T01:21:40.874 INFO:teuthology.orchestra.run.smithi167.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-19T01:21:40.877 INFO:teuthology.orchestra.run.smithi167.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-19T01:21:40.889 INFO:teuthology.orchestra.run.smithi167.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-19T01:21:40.889 INFO:teuthology.orchestra.run.smithi167.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-19T01:21:40.921 INFO:teuthology.orchestra.run.smithi167.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-19T01:21:40.921 INFO:teuthology.orchestra.run.smithi167.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-19T01:21:40.933 INFO:teuthology.orchestra.run.smithi167.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-19T01:21:40.950 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package flex. 2024-03-19T01:21:40.952 INFO:teuthology.orchestra.run.smithi167.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-19T01:21:40.957 INFO:teuthology.orchestra.run.smithi167.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-19T01:21:40.957 INFO:teuthology.orchestra.run.smithi167.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-19T01:21:40.958 INFO:teuthology.orchestra.run.smithi167.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-19T01:21:40.959 INFO:teuthology.orchestra.run.smithi167.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-19T01:21:40.982 INFO:teuthology.orchestra.run.smithi167.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-19T01:21:40.985 INFO:teuthology.orchestra.run.smithi167.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-19T01:21:41.056 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173660 files and directories currently installed.) 2024-03-19T01:21:41.061 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-19T01:21:41.098 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-19T01:21:41.322 INFO:teuthology.orchestra.run.smithi167.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-19T01:21:41.322 INFO:teuthology.orchestra.run.smithi167.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-19T01:21:41.322 INFO:teuthology.orchestra.run.smithi167.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-19T01:21:41.365 INFO:teuthology.orchestra.run.smithi167.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-19T01:21:41.365 INFO:teuthology.orchestra.run.smithi167.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-19T01:21:41.366 INFO:teuthology.orchestra.run.smithi167.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-19T01:21:41.386 INFO:teuthology.orchestra.run.smithi167.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-19T01:21:41.387 INFO:teuthology.orchestra.run.smithi167.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-19T01:21:41.665 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-19T01:21:41.786 INFO:teuthology.orchestra.run.smithi167.stdout:Preconfiguring packages ... 2024-03-19T01:21:41.846 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-19T01:21:41.953 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 54.6 MB in 1s (40.4 MB/s) 2024-03-19T01:21:42.102 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package flex. 2024-03-19T01:21:42.205 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173660 files and directories currently installed.) 2024-03-19T01:21:42.210 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-19T01:21:42.259 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-19T01:21:42.274 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:21:42.826 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-19T01:21:42.856 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package bison. 2024-03-19T01:21:42.963 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173733 files and directories currently installed.) 2024-03-19T01:21:42.969 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-19T01:21:43.007 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-19T01:21:43.017 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-19T01:21:43.342 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-19T01:21:43.371 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-19T01:21:43.401 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-19T01:21:43.485 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:21:43.745 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package dump. 2024-03-19T01:21:43.774 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-19T01:21:43.812 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking dump (0.4b47-3) ... 2024-03-19T01:21:44.034 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package bison. 2024-03-19T01:21:44.107 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package indent. 2024-03-19T01:21:44.124 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:44.138 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173733 files and directories currently installed.) 2024-03-19T01:21:44.144 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-19T01:21:44.165 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-19T01:21:44.194 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-19T01:21:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-19T01:21:44.427 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-19T01:21:44.460 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-19T01:21:44.604 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-19T01:21:44.631 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-19T01:21:44.670 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-19T01:21:44.773 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-19T01:21:44.800 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-19T01:21:44.839 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-19T01:21:45.091 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package dump. 2024-03-19T01:21:45.099 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-19T01:21:45.116 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-19T01:21:45.118 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-19T01:21:45.159 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-19T01:21:45.169 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking dump (0.4b47-3) ... 2024-03-19T01:21:45.469 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package indent. 2024-03-19T01:21:45.469 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-19T01:21:45.498 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:45.499 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-19T01:21:45.528 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-19T01:21:45.538 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-19T01:21:45.764 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-19T01:21:45.788 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-19T01:21:45.792 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-19T01:21:45.816 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-19T01:21:45.830 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-19T01:21:45.855 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-19T01:21:46.083 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libjson-perl. 2024-03-19T01:21:46.099 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-19T01:21:46.141 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-19T01:21:46.184 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-19T01:21:46.211 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-19T01:21:46.250 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-19T01:21:46.453 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-19T01:21:46.481 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-19T01:21:46.503 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-19T01:21:46.519 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-19T01:21:46.519 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-19T01:21:46.553 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-19T01:21:47.081 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-19T01:21:47.109 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-19T01:21:47.148 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-19T01:21:47.409 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-19T01:21:47.433 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-19T01:21:47.475 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-19T01:21:47.744 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libjson-perl. 2024-03-19T01:21:47.761 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-19T01:21:47.794 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-19T01:21:48.056 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-19T01:21:48.072 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-19T01:21:48.106 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-19T01:21:48.189 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-19T01:21:48.217 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-19T01:21:48.247 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:21:48.516 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-19T01:21:48.533 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-19T01:21:48.575 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:21:48.902 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-19T01:21:48.931 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-19T01:21:48.969 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:21:49.322 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-19T01:21:49.350 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-19T01:21:49.388 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-19T01:21:49.676 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-19T01:21:49.704 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-19T01:21:49.708 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-19T01:21:49.736 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-19T01:21:49.742 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:21:49.766 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-19T01:21:50.011 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-19T01:21:50.028 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-19T01:21:50.069 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:21:50.119 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-19T01:21:50.144 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:50.185 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:21:50.389 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-19T01:21:50.417 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-19T01:21:50.456 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:21:50.547 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-19T01:21:50.575 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:50.613 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:21:50.809 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-19T01:21:50.836 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-19T01:21:50.875 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-19T01:21:50.899 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-19T01:21:50.916 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:50.966 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:21:51.161 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-19T01:21:51.185 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-19T01:21:51.189 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-19T01:21:51.202 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-19T01:21:51.243 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:21:51.244 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-19T01:21:51.573 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-19T01:21:51.601 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:51.639 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:21:51.764 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-19T01:21:51.791 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-19T01:21:51.830 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-19T01:21:51.992 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-19T01:21:52.020 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:52.059 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:21:52.175 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package modemmanager. 2024-03-19T01:21:52.203 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-19T01:21:52.241 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:21:52.378 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-19T01:21:52.406 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:52.445 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:21:52.647 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-19T01:21:52.663 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-19T01:21:52.698 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:21:52.950 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package network-manager. 2024-03-19T01:21:52.979 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-19T01:21:53.025 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-19T01:21:53.218 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-19T01:21:53.246 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-19T01:21:53.276 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-19T01:21:53.612 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package modemmanager. 2024-03-19T01:21:53.640 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-19T01:21:53.674 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ppp. 2024-03-19T01:21:53.679 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:21:53.702 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-19T01:21:54.015 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-19T01:21:54.243 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package network-manager. 2024-03-19T01:21:54.271 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-19T01:21:54.311 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-19T01:21:54.337 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package pptp-linux. 2024-03-19T01:21:54.366 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-19T01:21:54.403 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-19T01:21:54.639 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-19T01:21:54.657 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-19T01:21:54.689 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-19T01:21:54.934 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ppp. 2024-03-19T01:21:54.962 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-19T01:21:54.993 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-19T01:21:55.022 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-19T01:21:55.059 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-client-common (238) ... 2024-03-19T01:21:55.223 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-19T01:21:55.363 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-19T01:21:55.392 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:55.429 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:21:55.572 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package pptp-linux. 2024-03-19T01:21:55.600 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-19T01:21:55.639 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-19T01:21:55.883 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-19T01:21:55.900 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-19T01:21:55.942 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-19T01:21:56.127 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-common. 2024-03-19T01:21:56.155 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-19T01:21:56.212 INFO:teuthology.orchestra.run.smithi007.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-19T01:21:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-common (238) ... 2024-03-19T01:21:56.244 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-19T01:21:56.273 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-19T01:21:56.311 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking postgresql-client-common (238) ... 2024-03-19T01:21:56.573 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-19T01:21:56.601 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:56.648 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:21:56.659 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-14. 2024-03-19T01:21:56.687 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:56.721 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:21:57.270 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package postgresql-common. 2024-03-19T01:21:57.299 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-19T01:21:57.356 INFO:teuthology.orchestra.run.smithi167.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-19T01:21:57.384 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking postgresql-common (238) ... 2024-03-19T01:21:57.853 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package postgresql-14. 2024-03-19T01:21:57.881 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:21:57.912 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql. 2024-03-19T01:21:57.923 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:21:57.942 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-19T01:21:57.979 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql (14+238) ... 2024-03-19T01:21:58.223 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-client. 2024-03-19T01:21:58.240 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-19T01:21:58.281 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-client (14+238) ... 2024-03-19T01:21:58.517 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-19T01:21:58.533 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-19T01:21:58.575 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-19T01:21:58.836 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package tcl8.6. 2024-03-19T01:21:58.865 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-19T01:21:58.894 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-19T01:21:59.163 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package tcl. 2024-03-19T01:21:59.192 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-19T01:21:59.237 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-19T01:21:59.315 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package postgresql. 2024-03-19T01:21:59.331 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-19T01:21:59.373 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking postgresql (14+238) ... 2024-03-19T01:21:59.583 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package wpasupplicant. 2024-03-19T01:21:59.584 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package postgresql-client. 2024-03-19T01:21:59.600 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-19T01:21:59.612 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-19T01:21:59.642 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking postgresql-client (14+238) ... 2024-03-19T01:21:59.650 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-19T01:21:59.886 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-19T01:21:59.902 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-19T01:21:59.944 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-19T01:21:59.979 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-19T01:21:59.996 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-19T01:22:00.029 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-19T01:22:00.247 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package tcl8.6. 2024-03-19T01:22:00.275 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-19T01:22:00.313 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-19T01:22:00.424 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-19T01:22:00.453 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-19T01:22:00.499 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-19T01:22:00.583 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package tcl. 2024-03-19T01:22:00.599 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-19T01:22:00.640 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-19T01:22:00.821 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-client-common (238) ... 2024-03-19T01:22:01.011 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package wpasupplicant. 2024-03-19T01:22:01.040 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-19T01:22:01.047 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-19T01:22:01.078 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-19T01:22:01.173 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-19T01:22:01.283 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:22:01.400 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-19T01:22:01.432 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-19T01:22:01.448 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-19T01:22:01.490 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-19T01:22:01.526 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-19T01:22:01.652 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:22:01.810 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-19T01:22:01.838 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-19T01:22:01.885 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-19T01:22:02.172 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-19T01:22:02.273 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up postgresql-client-common (238) ... 2024-03-19T01:22:02.275 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-19T01:22:02.450 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-19T01:22:02.585 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-19T01:22:02.661 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-19T01:22:02.662 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-19T01:22:02.711 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:22:02.862 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-19T01:22:02.988 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-19T01:22:03.114 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:22:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-19T01:22:03.279 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:22:03.405 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-19T01:22:03.531 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-19T01:22:03.599 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-19T01:22:03.632 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-19T01:22:03.678 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-19T01:22:03.733 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-19T01:22:03.851 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-19T01:22:04.079 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-19T01:22:04.079 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-19T01:22:04.523 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-19T01:22:04.589 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-19T01:22:04.633 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up dump (0.4b47-3) ... 2024-03-19T01:22:04.711 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-19T01:22:04.715 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:22:04.776 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:22:04.842 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-19T01:22:04.902 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-19T01:22:04.943 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-19T01:22:05.044 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-19T01:22:05.145 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-19T01:22:05.170 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-19T01:22:05.296 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-19T01:22:05.654 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-19T01:22:05.780 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-19T01:22:05.871 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-19T01:22:05.935 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-19T01:22:05.940 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-19T01:22:06.036 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up dump (0.4b47-3) ... 2024-03-19T01:22:06.066 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-19T01:22:06.138 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-19T01:22:06.204 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:22:06.209 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-19T01:22:06.310 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:22:06.330 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-19T01:22:06.428 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-19T01:22:06.456 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-19T01:22:06.596 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-client (14+238) ... 2024-03-19T01:22:06.607 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-19T01:22:06.722 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-19T01:22:06.742 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-19T01:22:06.832 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-19T01:22:06.902 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-19T01:22:07.003 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-19T01:22:07.171 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-19T01:22:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:07.207 INFO:teuthology.orchestra.run.smithi007.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-19T01:22:07.207 INFO:teuthology.orchestra.run.smithi007.stdout:which means they are currently configured by ifupdown: 2024-03-19T01:22:07.207 INFO:teuthology.orchestra.run.smithi007.stdout:- ens1f0 2024-03-19T01:22:07.207 INFO:teuthology.orchestra.run.smithi007.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-19T01:22:07.207 INFO:teuthology.orchestra.run.smithi007.stdout:remove their configuration from /etc/network/interfaces. 2024-03-19T01:22:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:07.297 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:22:07.324 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-19T01:22:07.423 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-19T01:22:07.591 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up postgresql-client (14+238) ... 2024-03-19T01:22:07.655 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-19T01:22:07.717 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-19T01:22:07.985 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-19T01:22:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:08.173 INFO:teuthology.orchestra.run.smithi167.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-19T01:22:08.173 INFO:teuthology.orchestra.run.smithi167.stdout:which means they are currently configured by ifupdown: 2024-03-19T01:22:08.173 INFO:teuthology.orchestra.run.smithi167.stdout:- enp3s0f1 2024-03-19T01:22:08.173 INFO:teuthology.orchestra.run.smithi167.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-19T01:22:08.173 INFO:teuthology.orchestra.run.smithi167.stdout:remove their configuration from /etc/network/interfaces. 2024-03-19T01:22:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:08.305 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-19T01:22:08.491 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:22:08.614 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-19T01:22:08.626 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-19T01:22:08.752 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-19T01:22:08.903 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-common (238) ... 2024-03-19T01:22:08.956 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-19T01:22:09.519 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:22:09.742 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-19T01:22:09.772 INFO:teuthology.orchestra.run.smithi007.stdout:Adding user postgres to group ssl-cert 2024-03-19T01:22:09.946 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-19T01:22:10.014 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:10.014 INFO:teuthology.orchestra.run.smithi007.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-19T01:22:10.172 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up postgresql-common (238) ... 2024-03-19T01:22:10.205 INFO:teuthology.orchestra.run.smithi007.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-19T01:22:10.205 INFO:teuthology.orchestra.run.smithi007.stdout:Removing obsolete dictionary files: 2024-03-19T01:22:10.788 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-19T01:22:11.172 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:22:11.506 INFO:teuthology.orchestra.run.smithi007.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-19T01:22:11.519 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-19T01:22:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-19T01:22:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:This user must also own the server process. 2024-03-19T01:22:11.566 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:The database cluster will be initialized with locale "en_US". 2024-03-19T01:22:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-19T01:22:11.567 INFO:teuthology.orchestra.run.smithi007.stdout:The default text search configuration will be set to "english". 2024-03-19T01:22:11.567 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:11.567 INFO:teuthology.orchestra.run.smithi007.stdout:Data page checksums are disabled. 2024-03-19T01:22:11.567 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:11.567 INFO:teuthology.orchestra.run.smithi007.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-19T01:22:11.567 INFO:teuthology.orchestra.run.smithi007.stdout:creating subdirectories ... ok 2024-03-19T01:22:11.567 INFO:teuthology.orchestra.run.smithi007.stdout:selecting dynamic shared memory implementation ... posix 2024-03-19T01:22:11.583 INFO:teuthology.orchestra.run.smithi007.stdout:selecting default max_connections ... 100 2024-03-19T01:22:11.603 INFO:teuthology.orchestra.run.smithi007.stdout:selecting default shared_buffers ... 128MB 2024-03-19T01:22:11.613 INFO:teuthology.orchestra.run.smithi007.stdout:selecting default time zone ... Etc/UTC 2024-03-19T01:22:11.614 INFO:teuthology.orchestra.run.smithi007.stdout:creating configuration files ... ok 2024-03-19T01:22:11.617 INFO:teuthology.orchestra.run.smithi167.stdout:Adding user postgres to group ssl-cert 2024-03-19T01:22:11.733 INFO:teuthology.orchestra.run.smithi007.stdout:running bootstrap script ... ok 2024-03-19T01:22:11.861 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:11.861 INFO:teuthology.orchestra.run.smithi167.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-19T01:22:12.056 INFO:teuthology.orchestra.run.smithi167.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-19T01:22:12.057 INFO:teuthology.orchestra.run.smithi167.stdout:Removing obsolete dictionary files: 2024-03-19T01:22:12.414 INFO:teuthology.orchestra.run.smithi007.stdout:performing post-bootstrap initialization ... ok 2024-03-19T01:22:12.593 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-19T01:22:12.953 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:22:13.041 INFO:teuthology.orchestra.run.smithi007.stdout:syncing data to disk ... ok 2024-03-19T01:22:13.261 INFO:teuthology.orchestra.run.smithi167.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-19T01:22:13.276 INFO:teuthology.orchestra.run.smithi167.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-19T01:22:13.324 INFO:teuthology.orchestra.run.smithi167.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-19T01:22:13.324 INFO:teuthology.orchestra.run.smithi167.stdout:This user must also own the server process. 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout:The database cluster will be initialized with locale "en_US". 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout:The default text search configuration will be set to "english". 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout:Data page checksums are disabled. 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout:creating subdirectories ... ok 2024-03-19T01:22:13.325 INFO:teuthology.orchestra.run.smithi167.stdout:selecting dynamic shared memory implementation ... posix 2024-03-19T01:22:13.340 INFO:teuthology.orchestra.run.smithi167.stdout:selecting default max_connections ... 100 2024-03-19T01:22:13.360 INFO:teuthology.orchestra.run.smithi167.stdout:selecting default shared_buffers ... 128MB 2024-03-19T01:22:13.370 INFO:teuthology.orchestra.run.smithi167.stdout:selecting default time zone ... Etc/UTC 2024-03-19T01:22:13.371 INFO:teuthology.orchestra.run.smithi167.stdout:creating configuration files ... ok 2024-03-19T01:22:13.446 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-19T01:22:13.487 INFO:teuthology.orchestra.run.smithi167.stdout:running bootstrap script ... ok 2024-03-19T01:22:14.108 INFO:teuthology.orchestra.run.smithi167.stdout:performing post-bootstrap initialization ... ok 2024-03-19T01:22:14.756 INFO:teuthology.orchestra.run.smithi167.stdout:syncing data to disk ... ok 2024-03-19T01:22:15.144 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-19T01:22:16.212 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:22:16.338 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-19T01:22:16.464 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up postgresql (14+238) ... 2024-03-19T01:22:16.573 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:22:16.674 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:22:16.916 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-19T01:22:16.917 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-19T01:22:17.460 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-19T01:22:17.804 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T01:22:17.918 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:22:17.939 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T01:22:18.019 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-19T01:22:18.145 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up postgresql (14+238) ... 2024-03-19T01:22:18.262 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:22:18.388 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:22:18.644 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-19T01:22:18.644 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-19T01:22:19.224 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-19T01:22:19.376 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-19T01:22:19.476 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T01:22:19.535 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T01:22:19.679 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T01:22:19.980 INFO:teuthology.orchestra.run.smithi007.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T01:22:19.986 DEBUG:teuthology.parallel:result is None 2024-03-19T01:22:20.816 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-19T01:22:20.916 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T01:22:21.419 INFO:teuthology.orchestra.run.smithi167.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T01:22:21.423 DEBUG:teuthology.parallel:result is None 2024-03-19T01:22:21.424 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:22:21.714 DEBUG:teuthology.orchestra.run.smithi007:> dpkg-query -W -f '${Version}' ceph 2024-03-19T01:22:21.734 INFO:teuthology.orchestra.run.smithi007.stdout:19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:22:21.735 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:22:21.735 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206-gb6ae2491-1jammy is installed. 2024-03-19T01:22:21.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:22:22.001 DEBUG:teuthology.orchestra.run.smithi167:> dpkg-query -W -f '${Version}' ceph 2024-03-19T01:22:22.021 INFO:teuthology.orchestra.run.smithi167.stdout:19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:22:22.022 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:22:22.022 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206-gb6ae2491-1jammy is installed. 2024-03-19T01:22:22.025 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-19T01:22:22.025 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:22.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T01:22:22.041 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:22.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T01:22:22.077 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-19T01:22:22.077 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:22.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T01:22:22.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T01:22:22.153 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:22.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T01:22:22.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T01:22:22.219 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-19T01:22:22.219 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:22.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T01:22:22.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T01:22:22.285 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:22.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T01:22:22.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T01:22:22.352 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-19T01:22:22.352 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:22.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T01:22:22.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T01:22:22.424 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:22.424 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T01:22:22.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T01:22:22.491 INFO:teuthology.run_tasks:Running task ceph... 2024-03-19T01:22:22.578 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-19T01:22:22.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2024-03-19T01:22:22.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /var/log/ceph 2024-03-19T01:22:22.594 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-19T01:22:22.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-19T01:22:22.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-19T01:22:22.647 INFO:tasks.ceph:Creating extra log directories... 2024-03-19T01:22:22.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-19T01:22:22.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-19T01:22:22.709 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-19T01:22:22.709 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-19T01:22:22.709 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-19T01:22:22.709 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:22.709 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-19T01:22:22.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-19T01:22:22.825 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:22.826 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-19T01:22:22.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-19T01:22:22.913 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-19T01:22:22.913 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-19T01:22:22.913 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297', 'branch': 'wip-batrick-testing-20240318.181317', 'description': 'fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7609297', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240318.181317'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240318.181317', 'sha1': 'e48b1f6c0608b2615e1890cc46286c3877d6e910'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3005, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'sleep_before_teardown': 0, 'subset': '433/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240318.181317', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph', 'suite_sha1': 'e48b1f6c0608b2615e1890cc46286c3877d6e910', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEk0CATx6EFmnq8G/eetGLUfRIFzePjKl1vLx1XTY7jtRWcro9dJkpYQ7AbkSZ/cqh/1TFSTXT4N26YAozbh5S4=', 'smithi167.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBAQ+hAwzR5bVVCCOh/YFbz5HEM/C7nGcxQonnlhWFYGHrOLA8v6OS5DUb26YGAKPfdWzQEgzrs0ki802j8QUmc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_cephfs_shell']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-18_21:05:49', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-19T01:22:22.914 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-19T01:22:22.916 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-19T01:22:22.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-03-19T01:22:22.962 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/run/ceph 2024-03-19T01:22:23.012 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:23.012 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T01:22:23.022 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T01:22:23.022 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-03-19T01:22:23.070 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T01:22:23.070 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:22:23.070 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 879 Links: 1 2024-03-19T01:22:23.070 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:22:23.070 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-19 01:16:06.267913919 +0000 2024-03-19T01:22:23.070 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-19 01:16:06.263913955 +0000 2024-03-19T01:22:23.070 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-19 01:16:06.263913955 +0000 2024-03-19T01:22:23.070 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-03-19T01:22:23.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T01:22:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-19T01:22:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-19T01:22:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000308081 s, 1.7 MB/s 2024-03-19T01:22:23.126 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T01:22:23.175 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-03-19T01:22:23.222 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T01:22:23.222 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:22:23.222 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 896 Links: 1 2024-03-19T01:22:23.222 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:22:23.222 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-19 01:16:07.023907208 +0000 2024-03-19T01:22:23.222 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-19 01:16:06.755909588 +0000 2024-03-19T01:22:23.222 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-19 01:16:06.755909588 +0000 2024-03-19T01:22:23.222 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-03-19T01:22:23.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T01:22:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-19T01:22:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-19T01:22:23.276 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000258471 s, 2.0 MB/s 2024-03-19T01:22:23.278 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T01:22:23.331 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-03-19T01:22:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T01:22:23.378 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:22:23.378 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 914 Links: 1 2024-03-19T01:22:23.378 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:22:23.378 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-19 01:16:07.507902911 +0000 2024-03-19T01:22:23.378 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-19 01:16:07.231905362 +0000 2024-03-19T01:22:23.378 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-19 01:16:07.231905362 +0000 2024-03-19T01:22:23.378 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-03-19T01:22:23.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T01:22:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-19T01:22:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-19T01:22:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000275974 s, 1.9 MB/s 2024-03-19T01:22:23.434 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T01:22:23.483 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-03-19T01:22:23.530 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T01:22:23.530 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:22:23.530 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 929 Links: 1 2024-03-19T01:22:23.530 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:22:23.530 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-19 01:16:07.751900745 +0000 2024-03-19T01:22:23.530 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-19 01:16:07.747900781 +0000 2024-03-19T01:22:23.530 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-19 01:16:07.747900781 +0000 2024-03-19T01:22:23.530 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-03-19T01:22:23.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T01:22:23.584 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-19T01:22:23.584 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-19T01:22:23.584 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000278574 s, 1.8 MB/s 2024-03-19T01:22:23.586 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T01:22:23.635 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:23.635 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:23.636 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T01:22:23.641 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T01:22:23.641 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2024-03-19T01:22:23.688 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T01:22:23.688 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:22:23.689 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-03-19T01:22:23.689 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:22:23.689 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-03-19 01:16:02.392766432 +0000 2024-03-19T01:22:23.689 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-03-19 01:16:02.116768895 +0000 2024-03-19T01:22:23.689 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-03-19 01:16:02.116768895 +0000 2024-03-19T01:22:23.689 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2024-03-19T01:22:23.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T01:22:23.743 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-03-19T01:22:23.743 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-03-19T01:22:23.743 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000453779 s, 1.1 MB/s 2024-03-19T01:22:23.744 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T01:22:23.793 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2024-03-19T01:22:23.840 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T01:22:23.840 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:22:23.840 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-03-19T01:22:23.840 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:22:23.840 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-03-19 01:16:02.904761865 +0000 2024-03-19T01:22:23.840 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-03-19 01:16:02.644764185 +0000 2024-03-19T01:22:23.840 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-03-19 01:16:02.644764185 +0000 2024-03-19T01:22:23.840 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2024-03-19T01:22:23.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T01:22:23.895 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-03-19T01:22:23.895 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-03-19T01:22:23.895 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000405488 s, 1.3 MB/s 2024-03-19T01:22:23.896 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T01:22:23.945 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2024-03-19T01:22:23.992 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T01:22:23.992 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:22:23.992 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 924 Links: 1 2024-03-19T01:22:23.992 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:22:23.993 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-03-19 01:16:03.396757476 +0000 2024-03-19T01:22:23.993 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-03-19 01:16:03.148759689 +0000 2024-03-19T01:22:23.993 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-03-19 01:16:03.148759689 +0000 2024-03-19T01:22:23.993 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2024-03-19T01:22:23.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T01:22:24.047 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-03-19T01:22:24.047 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-03-19T01:22:24.047 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000421317 s, 1.2 MB/s 2024-03-19T01:22:24.048 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T01:22:24.101 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2024-03-19T01:22:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T01:22:24.148 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:22:24.148 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 944 Links: 1 2024-03-19T01:22:24.148 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:22:24.148 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-03-19 01:16:03.880753158 +0000 2024-03-19T01:22:24.149 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-03-19 01:16:03.620755478 +0000 2024-03-19T01:22:24.149 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-03-19 01:16:03.620755478 +0000 2024-03-19T01:22:24.149 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2024-03-19T01:22:24.149 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T01:22:24.203 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-03-19T01:22:24.203 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-03-19T01:22:24.203 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000392917 s, 1.3 MB/s 2024-03-19T01:22:24.205 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T01:22:24.253 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:24.253 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-19T01:22:24.254 INFO:tasks.ceph:Generating config... 2024-03-19T01:22:24.255 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-19T01:22:24.255 INFO:tasks.ceph:[client] debug client = 20 2024-03-19T01:22:24.255 INFO:tasks.ceph:[client] debug ms = 1 2024-03-19T01:22:24.255 INFO:tasks.ceph:[client] pid file = 2024-03-19T01:22:24.255 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-19T01:22:24.255 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-19T01:22:24.255 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-19T01:22:24.255 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-19T01:22:24.255 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-19T01:22:24.255 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-19T01:22:24.255 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-19T01:22:24.255 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-19T01:22:24.255 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-19T01:22:24.256 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-19T01:22:24.256 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-19T01:22:24.256 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-19T01:22:24.256 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-19T01:22:24.256 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-19T01:22:24.256 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-19T01:22:24.256 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-19T01:22:24.256 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-19T01:22:24.256 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-19T01:22:24.256 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-19T01:22:24.256 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-19T01:22:24.256 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-19T01:22:24.256 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-19T01:22:24.256 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-19T01:22:24.256 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-19T01:22:24.256 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-19T01:22:24.257 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-19T01:22:24.257 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-19T01:22:24.257 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-19T01:22:24.257 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-19T01:22:24.257 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-19T01:22:24.257 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-19T01:22:24.257 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-19T01:22:24.257 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-19T01:22:24.257 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-19T01:22:24.257 INFO:tasks.ceph:Setting up mon.a... 2024-03-19T01:22:24.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-19T01:22:24.293 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2024-03-19T01:22:24.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-19T01:22:24.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T01:22:24.445 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.7'), ('mon.b', '172.21.15.167'), ('mon.c', '[v2:172.21.15.167:3301,v1:172.21.15.167:6790]')] 2024-03-19T01:22:24.445 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7,172.21.15.167,[v2:172.21.15.167:3301,v1:172.21.15.167:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.b': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-19T01:22:24.446 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:24.446 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-19T01:22:24.494 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --add b 172.21.15.167 --addv c '[v2:172.21.15.167:3301,v1:172.21.15.167:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:24.580 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid dafc8aaa-17d4-48ac-88de-2f029fdf817b 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:fsid dafc8aaa-17d4-48ac-88de-2f029fdf817b 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2024-03-19T01:22:24.583314+0000 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-03-19T01:22:24.583314+0000 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 17 (quincy) 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:2: [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] mon.c 2024-03-19T01:22:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-19T01:22:24.583 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-19T01:22:24.630 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID dafc8aaa-17d4-48ac-88de-2f029fdf817b... 2024-03-19T01:22:24.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-19T01:22:24.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-19T01:22:24.702 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-03-19T01:22:24.702 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-03-19T01:22:24.702 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-19T01:22:24.702 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2024-03-19T01:22:24.702 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.703 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.704 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = "172.21.15.7,172.21.15.167,[v2:172.21.15.167:3301,v1:172.21.15.167:6790]" 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = dafc8aaa-17d4-48ac-88de-2f029fdf817b 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.705 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-03-19T01:22:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.707 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: bdev async discard = True 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: bdev enable discard = True 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore allocator = bitmap 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 1/20 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 1/20 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 4/10 2024-03-19T01:22:24.708 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: osd op complaint time = 180 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-03-19T01:22:24.709 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.710 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: mon op complaint time = 120 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-03-19T01:22:24.711 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: client mount timeout = 600 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: debug client = 20 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = "" 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: rados mon op timeout = 900 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout: rados osd op timeout = 900 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.b] 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.c] 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.a] 2024-03-19T01:22:24.712 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.b] 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout:[mds] 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds = 20 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds balancer = 20 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: mds debug frag = True 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: mds debug scatterstat = True 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: mds op complaint time = 180 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: mds verify scatter = True 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: osd op complaint time = 180 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: rados mon op timeout = 900 2024-03-19T01:22:24.713 INFO:teuthology.orchestra.run.smithi007.stdout: rados osd op timeout = 900 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout:[global] 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: chdir = "" 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: auth supported = cephx 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: filestore xattr use omap = true 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: mon clock drift allowed = 1.000 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: osd crush chooseleaf type = 0 2024-03-19T01:22:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: auth debug = true 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on old message = true 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on bug = true 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon pg warn max object skew = 0 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: # disable pg_autoscaler by default for new pools 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default size = 2 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow primary affinity = true 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow pg remap = true 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on legacy crush tunables = false 2024-03-19T01:22:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on crush straw calc version zero = false 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on no sortbitwise = false 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on osd down out interval zero = false 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on too few osds = false 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: mon_allow_pool_size_one = true 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: osd default data pool replay window = 5 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: mon allow pool delete = true 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: mon cluster log file level = debug 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: debug asserts on shutdown = true 2024-03-19T01:22:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: mon health detail to clog = false 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: mon host = "172.21.15.7,172.21.15.167,[v2:172.21.15.167:3301,v1:172.21.15.167:6790]" 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: fsid = dafc8aaa-17d4-48ac-88de-2f029fdf817b 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout:[osd] 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd journal size = 100 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub load threshold = 5.0 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub max interval = 600 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock profile = high_recovery_ops 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd recover clone overlap = true 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd recovery max chunk = 1048576 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug shutdown = true 2024-03-19T01:22:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug op order = true 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify stray on activate = true 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug trim objects = true 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: osd open classes on start = true 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug pg log writeout = true 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: osd deep scrub update digest min age = 30 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: osd map max advance = 10 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: journal zero on create = true 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: filestore ondisk finisher threads = 3 2024-03-19T01:22:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: filestore apply finisher threads = 3 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: bdev debug aio = true 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug misdirected ops = true 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: bdev async discard = True 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: bdev enable discard = True 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore allocator = bitmap 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore block size = 96636764160 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore fsck on mount = True 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluefs = 1/20 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluestore = 1/20 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug osd = 20 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: debug rocksdb = 4/10 2024-03-19T01:22:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd backfillfull_ratio = 0.85 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd full ratio = 0.9 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd nearfull ratio = 0.8 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: osd failsafe full ratio = 0.95 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: osd objectstore = bluestore 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: osd op complaint time = 180 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr] 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: debug mgr = 20 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: mgr/telemetry/nag = false 2024-03-19T01:22:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout:[mon] 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: debug paxos = 20 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: mon data avail warn = 5 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: mon mgr mkfs grace = 240 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd reporter subtree level = osd 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd prime pg temp = true 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-19T01:22:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: auth mon ticket ttl = 660 # 11m 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: auth service ticket ttl = 240 # 4m 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: # don't complain about insecure global_id in the test suite 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: # 1m isn't quite enough 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon_down_mkfs_grace = 2m 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_filestore_osds = false 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: mon op complaint time = 120 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout:[client] 2024-03-19T01:22:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: rgw cache enabled = true 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable ops log = true 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable usage log = true 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: client mount timeout = 600 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: debug client = 20 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = "" 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: rados mon op timeout = 900 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: rados osd op timeout = 900 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.a] 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.b] 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.c] 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.a] 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.b] 2024-03-19T01:22:24.723 INFO:teuthology.orchestra.run.smithi167.stdout:[mds] 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: debug mds = 20 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: debug mds balancer = 20 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: mds debug frag = True 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: mds debug scatterstat = True 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: mds op complaint time = 180 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: mds verify scatter = True 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: osd op complaint time = 180 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: rados mon op timeout = 900 2024-03-19T01:22:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: rados osd op timeout = 900 2024-03-19T01:22:24.725 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-19T01:22:24.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-19T01:22:24.819 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-19T01:22:24.820 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:24.820 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-19T01:22:24.825 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:24.825 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-19T01:22:24.877 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:22:24.877 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:24.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-19T01:22:24.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T01:22:24.942 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:24.942 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:24.990 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:22:24.990 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:24.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-19T01:22:24.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T01:22:25.012 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:25.013 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:25.060 INFO:tasks.ceph:Setting up mon nodes... 2024-03-19T01:22:25.060 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-19T01:22:25.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-19T01:22:25.104 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-19T01:22:25.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-19T01:22:25.150 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-19T01:22:25.156 INFO:tasks.ceph:Setting up mds nodes... 2024-03-19T01:22:25.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-19T01:22:25.200 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-19T01:22:25.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-19T01:22:25.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-19T01:22:25.305 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-19T01:22:25.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-19T01:22:25.363 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-19T01:22:25.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-19T01:22:25.399 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-19T01:22:25.412 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-19T01:22:25.412 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-19T01:22:25.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-19T01:22:25.465 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:25.465 INFO:tasks.ceph:role: osd.0 2024-03-19T01:22:25.465 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:22:25.466 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-19T01:22:25.522 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:22:25.522 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:22:25.522 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:22:25.522 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:22:25.522 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:22:25.523 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:22:25.523 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:22:25.523 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:22:25.523 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:22:25.523 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:22:25.549 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-19T01:22:25.551 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-19T01:22:25.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-19T01:22:25.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-19T01:22:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:22:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:22:25.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-19T01:22:25.721 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:25.721 INFO:tasks.ceph:role: osd.1 2024-03-19T01:22:25.722 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:22:25.722 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-19T01:22:25.776 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:22:25.777 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:22:25.777 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:22:25.777 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:22:25.777 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:22:25.777 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:22:25.777 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:22:25.777 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:22:25.777 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:22:25.777 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:22:25.800 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-19T01:22:25.801 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-19T01:22:25.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-19T01:22:25.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-19T01:22:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:22:25.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:22:25.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-19T01:22:25.971 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:25.971 INFO:tasks.ceph:role: osd.2 2024-03-19T01:22:25.971 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:22:25.971 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-19T01:22:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:22:26.026 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:22:26.026 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:22:26.026 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:22:26.026 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:22:26.026 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:22:26.026 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:22:26.026 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:22:26.026 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:22:26.026 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:22:26.050 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-19T01:22:26.051 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-19T01:22:26.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-19T01:22:26.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-19T01:22:26.164 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:22:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:22:26.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-19T01:22:26.222 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:26.222 INFO:tasks.ceph:role: osd.3 2024-03-19T01:22:26.222 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:22:26.222 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:22:26.277 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:22:26.300 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-19T01:22:26.302 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-19T01:22:26.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-19T01:22:26.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-19T01:22:26.416 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:22:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:22:26.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:26.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:26.674+0000 7fe6ae23b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-19T01:22:26.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:26.674+0000 7fe6ae23b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-19T01:22:26.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:26.674+0000 7fe6ae23b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T01:22:26.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:26.674+0000 7fe6ae23b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T01:22:26.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:26.674+0000 7fe6ae23b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T01:22:26.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:26.674+0000 7fe6ae23b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-19T01:22:29.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-19T01:22:29.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:29.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:29.734+0000 7f32853b07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-19T01:22:29.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:29.734+0000 7f32853b07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-19T01:22:29.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:29.734+0000 7f32853b07c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T01:22:29.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:29.734+0000 7f32853b07c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T01:22:29.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:29.734+0000 7f32853b07c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T01:22:29.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:29.734+0000 7f32853b07c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-19T01:22:32.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-19T01:22:32.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:32.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:32.834+0000 7f563a0657c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-19T01:22:32.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:32.834+0000 7f563a0657c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-19T01:22:32.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:32.834+0000 7f563a0657c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T01:22:32.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:32.834+0000 7f563a0657c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T01:22:32.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:32.834+0000 7f563a0657c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T01:22:32.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:32.838+0000 7f563a0657c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-19T01:22:35.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-19T01:22:35.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:35.930+0000 7f7bd9a277c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-19T01:22:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:35.930+0000 7f7bd9a277c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-19T01:22:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:35.930+0000 7f7bd9a277c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T01:22:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:35.930+0000 7f7bd9a277c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T01:22:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:35.930+0000 7f7bd9a277c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T01:22:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:35.930+0000 7f7bd9a277c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-19T01:22:38.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-19T01:22:38.966 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-19T01:22:38.981 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:38.981 INFO:tasks.ceph:role: osd.4 2024-03-19T01:22:38.981 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:22:38.981 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-19T01:22:39.043 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:22:39.043 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:22:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:22:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:22:39.044 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:22:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:22:39.044 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:22:39.044 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:22:39.044 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:22:39.044 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:22:39.502 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-03-19T01:22:39.504 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-03-19T01:22:39.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-19T01:22:39.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-19T01:22:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:22:39.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:22:39.632 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-19T01:22:39.687 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:39.688 INFO:tasks.ceph:role: osd.5 2024-03-19T01:22:39.688 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:22:39.688 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-19T01:22:39.751 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:22:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:22:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:22:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:22:39.751 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:22:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:22:39.752 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:22:39.752 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:22:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:22:39.752 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:22:40.224 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-03-19T01:22:40.226 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-03-19T01:22:40.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-19T01:22:40.379 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-19T01:22:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:22:40.391 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:22:40.391 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-19T01:22:40.447 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:40.447 INFO:tasks.ceph:role: osd.6 2024-03-19T01:22:40.447 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:22:40.448 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:22:40.513 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:22:40.989 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-03-19T01:22:40.990 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-03-19T01:22:40.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-19T01:22:41.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-19T01:22:41.156 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:22:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:22:41.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-19T01:22:41.212 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:22:41.212 INFO:tasks.ceph:role: osd.7 2024-03-19T01:22:41.212 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:22:41.213 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:22:41.276 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:22:41.697 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-03-19T01:22:41.699 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-03-19T01:22:41.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-19T01:22:41.790 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-19T01:22:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:22:41.840 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:22:41.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:41.950 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:41.947+0000 7fe12efae7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-19T01:22:41.955 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:41.955+0000 7fe12efae7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-19T01:22:41.956 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:41.955+0000 7fe12efae7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T01:22:41.956 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:41.955+0000 7fe12efae7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T01:22:41.956 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:41.955+0000 7fe12efae7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T01:22:41.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:41.955+0000 7fe12efae7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-19T01:22:44.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-19T01:22:44.955 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:45.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:45.035+0000 7fe793a307c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-19T01:22:45.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:45.035+0000 7fe793a307c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-19T01:22:45.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:45.035+0000 7fe793a307c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T01:22:45.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:45.035+0000 7fe793a307c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T01:22:45.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:45.035+0000 7fe793a307c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T01:22:45.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:45.035+0000 7fe793a307c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-19T01:22:48.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-19T01:22:48.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:48.145 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:48.143+0000 7fe8cebd67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-19T01:22:48.145 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:48.143+0000 7fe8cebd67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-19T01:22:48.145 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:48.143+0000 7fe8cebd67c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T01:22:48.145 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:48.143+0000 7fe8cebd67c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T01:22:48.146 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:48.143+0000 7fe8cebd67c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T01:22:48.146 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:48.143+0000 7fe8cebd67c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-19T01:22:51.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-19T01:22:51.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:51.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:51.262+0000 7fb7b80677c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-19T01:22:51.266 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:51.262+0000 7fb7b80677c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-19T01:22:51.266 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:51.262+0000 7fb7b80677c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T01:22:51.266 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:51.262+0000 7fb7b80677c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T01:22:51.267 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:51.262+0000 7fb7b80677c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T01:22:51.267 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:22:51.262+0000 7fb7b80677c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-19T01:22:52.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:22:52.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:22:54.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-19T01:22:54.260 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-19T01:22:54.261 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:54.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-19T01:22:54.275 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:54.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-19T01:22:54.329 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:54.329 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-19T01:22:54.390 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:54.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-19T01:22:54.445 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:54.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-19T01:22:54.502 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:54.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-19T01:22:54.558 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:54.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-19T01:22:54.572 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:54.572 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-19T01:22:54.627 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:54.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-19T01:22:54.683 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:54.683 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-19T01:22:54.735 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:54.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-19T01:22:54.791 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:22:54.791 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-19T01:22:54.846 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:22:54.847 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-19T01:22:54.853 INFO:tasks.ceph:Adding keys to all mons... 2024-03-19T01:22:54.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-19T01:22:54.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-19T01:22:54.908 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.y] 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDR6PhlSe9fBhAAvNqkKFNocL9xCVoNfTGX0Q== 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.a] 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDR6PhlK/AUDBAA94pHbItEQ9o86j85imfdIA== 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDS6PhlNphlKBAAyunkfFzvQpgLzXiQph4oyQ== 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDV6Phli9EGLBAAWnUfdc3mT1J5vvQGjw4FAw== 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDY6PhlmhQFMhAAwo0If4nQ49q643BAJK+Z+A== 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.3] 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDb6PhlB5a5NxAAArM2TyZLtBVtSY0i5regNg== 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2024-03-19T01:22:54.909 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDR6PhlET0KCRAA70G3XCxdRYYJWv1Yw5u3dg== 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.b] 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDR6PhljWJJEhAAhzDx2ziIc1Xqn2GMaQuJSw== 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.4] 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDh6Phl1Su8OBAAeE9KMWzAW6lIfvv1VLrOQw== 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.5] 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDl6PhljbpUAhAAC7o7ETDafwt1BJxzituBxQ== 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.6] 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDo6Phl4qrACBAAmjBNXyYzX7mEGR0ciRoHTA== 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.7] 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDr6PhlpJXxDxAAB17KJ8tzGewusYgGNmg5nA== 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-03-19T01:22:54.910 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDR6Phl6xj5FxAAeVMK9nlnMPrcDkxprwQY2w== 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.y] 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDR6PhlSe9fBhAAvNqkKFNocL9xCVoNfTGX0Q== 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.a] 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDR6PhlK/AUDBAA94pHbItEQ9o86j85imfdIA== 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.0] 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDS6PhlNphlKBAAyunkfFzvQpgLzXiQph4oyQ== 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.1] 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDV6Phli9EGLBAAWnUfdc3mT1J5vvQGjw4FAw== 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.2] 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDY6PhlmhQFMhAAwo0If4nQ49q643BAJK+Z+A== 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.3] 2024-03-19T01:22:54.912 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDb6PhlB5a5NxAAArM2TyZLtBVtSY0i5regNg== 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.x] 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDR6PhlET0KCRAA70G3XCxdRYYJWv1Yw5u3dg== 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.b] 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDR6PhljWJJEhAAhzDx2ziIc1Xqn2GMaQuJSw== 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.4] 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDh6Phl1Su8OBAAeE9KMWzAW6lIfvv1VLrOQw== 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.5] 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDl6PhljbpUAhAAC7o7ETDafwt1BJxzituBxQ== 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.6] 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDo6Phl4qrACBAAmjBNXyYzX7mEGR0ciRoHTA== 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.7] 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDr6PhlpJXxDxAAB17KJ8tzGewusYgGNmg5nA== 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout:[client.0] 2024-03-19T01:22:54.913 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQDR6Phl6xj5FxAAeVMK9nlnMPrcDkxprwQY2w== 2024-03-19T01:22:54.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T01:22:54.955 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T01:22:55.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:22:55.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:22:55.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.218 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T01:22:55.390 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T01:22:55.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:22:55.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:22:55.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.698 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:22:55.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T01:22:55.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T01:22:55.925 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-19T01:22:55.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-19T01:22:55.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T01:22:56.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-19T01:22:56.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-19T01:22:56.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T01:22:56.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-19T01:22:56.185 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-19T01:22:56.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T01:22:56.326 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-19T01:22:56.336 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:56.339 DEBUG:teuthology.orchestra.run.smithi167:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:22:56.383 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-19T01:22:56.384 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-19T01:22:56.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-19T01:22:56.386 INFO:tasks.ceph.mon.a:Started 2024-03-19T01:22:56.387 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-19T01:22:56.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-19T01:22:56.429 INFO:tasks.ceph.mon.b:Started 2024-03-19T01:22:56.429 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-19T01:22:56.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-19T01:22:56.431 INFO:tasks.ceph.mon.c:Started 2024-03-19T01:22:56.431 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-19T01:22:56.432 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-19T01:22:56.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-19T01:22:56.434 INFO:tasks.ceph.mgr.y:Started 2024-03-19T01:22:56.434 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-19T01:22:56.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-19T01:22:56.437 INFO:tasks.ceph.mgr.x:Started 2024-03-19T01:22:56.437 INFO:tasks.ceph:Setting crush tunables to default 2024-03-19T01:22:56.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2024-03-19T01:22:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f131ebaf640 1 Processor -- start 2024-03-19T01:22:56.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f131ebaf640 1 -- start start 2024-03-19T01:22:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f131ebaf640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f131ebaf640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131814a820 con 0x7f131814fd80 2024-03-19T01:22:56.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f131ebaf640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13180afee0 con 0x7f13180b5780 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f131ebaf640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f13180b6250 con 0x7f1318149e50 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f1317fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f1317fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44164/0 (socket says 172.21.15.7:44164) 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f131c94b640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f13180b5780 0x7f131814aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:46048/0 (socket says 172.21.15.7:46048) 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 learned_addr learned my addr 172.21.15.7:0/1202358168 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f131d14c640 1 --1- 172.21.15.7:0/1202358168 >> v1:172.21.15.7:6789/0 conn(0x7f131814fd80 0x7f1318152170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34894/0 (socket says 172.21.15.7:34894) 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 msgr2=0x7f131814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 msgr2=0x7f131814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:22:56.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.554+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:22:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.758+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.758+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 msgr2=0x7f131814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:22:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.758+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 msgr2=0x7f131814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:22:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.758+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:22:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:56.758+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:22:57.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.162+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:57.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.162+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 msgr2=0x7f131814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:22:57.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.162+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 msgr2=0x7f131814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:22:57.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.162+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:22:57.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.162+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:22:57.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.965+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:57.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.965+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 msgr2=0x7f131814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:22:57.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.965+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 msgr2=0x7f131814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:22:57.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.965+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:22:57.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:57.965+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:22:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 msgr2=0x7f131814a2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f13167fc640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.7:3300/0 conn(0x7f1318149e50 0x7f131814a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 >> v1:172.21.15.167:6789/0 conn(0x7f13180b5780 legacy=0x7f131814aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 >> v1:172.21.15.7:6789/0 conn(0x7f131814fd80 legacy=0x7f1318152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f13167fc640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f13167fc640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f13180b6250 con 0x7f12f80036f0 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f12f8008d80 con 0x7f12f8006190 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 msgr2=0x7f12f8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 msgr2=0x7f12f8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:22:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:22:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f131c94b640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 msgr2=0x7f12f8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-19T01:22:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f131c94b640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 msgr2=0x7f12f8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:22:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:22:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:22:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.553+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:22:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f131c94b640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 msgr2=0x7f12f8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:22:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f131c94b640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 msgr2=0x7f12f8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:22:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:22:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:22:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 msgr2=0x7f12f8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-19T01:22:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 msgr2=0x7f12f8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:22:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:22:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:22:59.757+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:23:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 msgr2=0x7f12f8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:23:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 msgr2=0x7f12f8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:23:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f131c94b640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 msgr2=0x7f12f8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-19T01:23:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f131c94b640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 msgr2=0x7f12f8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:23:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:23:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:23:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:23:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.161+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f131c94b640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 msgr2=0x7f12f8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f131c94b640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 msgr2=0x7f12f8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 msgr2=0x7f12f8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 msgr2=0x7f12f8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f131c94b640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:23:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:00.965+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:23:02.110 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:02.109+0000 7f198c2e3180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T01:23:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 msgr2=0x7f12f8008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:23:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13167fc640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8006190 0x7f12f8008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 msgr2=0x7f12f8005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:23:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13167fc640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f80036f0 0x7f12f8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f131d14c640 1 -- 172.21.15.7:0/1202358168 reap_dead start 2024-03-19T01:23:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13167fc640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f8006620 0x7f12f8003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131814b170 con 0x7f131814fd80 2024-03-19T01:23:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f8005d70 con 0x7f12f8004010 2024-03-19T01:23:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13167fc640 1 -- 172.21.15.7:0/1202358168 --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f13180b6250 con 0x7f12f8006620 2024-03-19T01:23:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> v2:172.21.15.167:3300/0 conn(0x7f12f8006620 0x7f12f8003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13177fe640 1 -- 172.21.15.7:0/1202358168 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2196390313 0 0) 0x7f131814b170 con 0x7f131814fd80 2024-03-19T01:23:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f13177fe640 1 -- 172.21.15.7:0/1202358168 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1304003530 con 0x7f131814fd80 2024-03-19T01:23:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v1:172.21.15.167:6789/0 conn(0x7f12f8004010 legacy=0x7f12f80044c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.553+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 >> v1:172.21.15.7:6789/0 conn(0x7f131814fd80 legacy=0x7f12f800b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.557+0000 7f1317fff640 1 -- 172.21.15.7:0/1202358168 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f80109e0 con 0x7f12f8006620 2024-03-19T01:23:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f1317fff640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12f8006620 0x7f12f8003ad0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1308009100 tx=0x7f130801de30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1189e97a24573ee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f13177fe640 1 -- 172.21.15.7:0/1202358168 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13080b3070 con 0x7f12f8006620 2024-03-19T01:23:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f13177fe640 1 -- 172.21.15.7:0/1202358168 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13080a6bf0 con 0x7f12f8006620 2024-03-19T01:23:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f13177fe640 1 -- 172.21.15.7:0/1202358168 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130800f040 con 0x7f12f8006620 2024-03-19T01:23:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f131ebaf640 1 -- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12f8006620 msgr2=0x7f12f8003ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f131ebaf640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12f8006620 0x7f12f8003ad0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1308009100 tx=0x7f130801de30 comp rx=0 tx=0).stop 2024-03-19T01:23:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f131ebaf640 1 -- 172.21.15.7:0/1202358168 shutdown_connections 2024-03-19T01:23:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f131ebaf640 1 --2- 172.21.15.7:0/1202358168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12f8006620 0x7f12f8003ad0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f131ebaf640 1 -- 172.21.15.7:0/1202358168 >> 172.21.15.7:0/1202358168 conn(0x7f13180b6610 msgr2=0x7f13180b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f131ebaf640 1 -- 172.21.15.7:0/1202358168 shutdown_connections 2024-03-19T01:23:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.561+0000 7f131ebaf640 1 -- 172.21.15.7:0/1202358168 wait complete. 2024-03-19T01:23:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131ebaf640 1 Processor -- start 2024-03-19T01:23:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131ebaf640 1 -- start start 2024-03-19T01:23:02.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131ebaf640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8004010 0x7f1318148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:02.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131ebaf640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12f8006620 0x7f1318147e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131ebaf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f131814fd80 0x7f13181e3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131ebaf640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f131806d2a0 con 0x7f131814fd80 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131ebaf640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f131814a670 con 0x7f12f8006620 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131ebaf640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f13180af9c0 con 0x7f12f8004010 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131d14c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f131814fd80 0x7f13181e3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131d14c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f131814fd80 0x7f13181e3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40270/0 (socket says 172.21.15.7:40270) 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131d14c640 1 -- 172.21.15.7:0/1318149834 learned_addr learned my addr 172.21.15.7:0/1318149834 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f1317fff640 1 --2- 172.21.15.7:0/1318149834 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12f8006620 0x7f1318147e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131c94b640 1 --2- 172.21.15.7:0/1318149834 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8004010 0x7f1318148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131d14c640 1 -- 172.21.15.7:0/1318149834 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8004010 msgr2=0x7f1318148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131d14c640 1 --2- 172.21.15.7:0/1318149834 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8004010 0x7f1318148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131d14c640 1 -- 172.21.15.7:0/1318149834 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12f8006620 msgr2=0x7f1318147e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131d14c640 1 --2- 172.21.15.7:0/1318149834 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12f8006620 0x7f1318147e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.565+0000 7f131d14c640 1 -- 172.21.15.7:0/1318149834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13181483c0 con 0x7f131814fd80 2024-03-19T01:23:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.569+0000 7f131d14c640 1 --2- 172.21.15.7:0/1318149834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f131814fd80 0x7f13181e3600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f13180b41d0 tx=0x7f131000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.569+0000 7f13157fa640 1 -- 172.21.15.7:0/1318149834 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1310017070 con 0x7f131814fd80 2024-03-19T01:23:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.569+0000 7f13157fa640 1 -- 172.21.15.7:0/1318149834 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13100070d0 con 0x7f131814fd80 2024-03-19T01:23:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.569+0000 7f13157fa640 1 -- 172.21.15.7:0/1318149834 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13100074c0 con 0x7f131814fd80 2024-03-19T01:23:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.569+0000 7f131ebaf640 1 -- 172.21.15.7:0/1318149834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13180b0d60 con 0x7f131814fd80 2024-03-19T01:23:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.569+0000 7f131ebaf640 1 -- 172.21.15.7:0/1318149834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13180b1c50 con 0x7f131814fd80 2024-03-19T01:23:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.569+0000 7f13157fa640 1 -- 172.21.15.7:0/1318149834 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f131001a070 con 0x7f131814fd80 2024-03-19T01:23:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.569+0000 7f13157fa640 1 -- 172.21.15.7:0/1318149834 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1310012400 con 0x7f131814fd80 2024-03-19T01:23:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.569+0000 7f131ebaf640 1 -- 172.21.15.7:0/1318149834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1318075570 con 0x7f131814fd80 2024-03-19T01:23:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.585+0000 7f13157fa640 1 -- 172.21.15.7:0/1318149834 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1310021020 con 0x7f131814fd80 2024-03-19T01:23:02.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.645+0000 7f131ebaf640 1 -- 172.21.15.7:0/1318149834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f13180720a0 con 0x7f131814fd80 2024-03-19T01:23:02.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f13157fa640 1 -- 172.21.15.7:0/1318149834 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f13100076b0 con 0x7f131814fd80 2024-03-19T01:23:02.652 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-03-19T01:23:02.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f131ebaf640 1 -- 172.21.15.7:0/1318149834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f131814fd80 msgr2=0x7f13181e3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:02.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f131ebaf640 1 --2- 172.21.15.7:0/1318149834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f131814fd80 0x7f13181e3600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f13180b41d0 tx=0x7f131000c960 comp rx=0 tx=0).stop 2024-03-19T01:23:02.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f131ebaf640 1 -- 172.21.15.7:0/1318149834 shutdown_connections 2024-03-19T01:23:02.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f131ebaf640 1 --2- 172.21.15.7:0/1318149834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f131814fd80 0x7f13181e3600 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:02.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f131ebaf640 1 --2- 172.21.15.7:0/1318149834 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12f8006620 0x7f1318147e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:02.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f131ebaf640 1 --2- 172.21.15.7:0/1318149834 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12f8004010 0x7f1318148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:02.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f131ebaf640 1 -- 172.21.15.7:0/1318149834 >> 172.21.15.7:0/1318149834 conn(0x7f13180b6610 msgr2=0x7f131814ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:02.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f131ebaf640 1 -- 172.21.15.7:0/1318149834 shutdown_connections 2024-03-19T01:23:02.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:02.649+0000 7f131ebaf640 1 -- 172.21.15.7:0/1318149834 wait complete. 2024-03-19T01:23:02.670 INFO:tasks.ceph:check_enable_crimson: False 2024-03-19T01:23:02.670 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-19T01:23:02.671 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:23:02.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-19T01:23:02.683 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:23:02.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-19T01:23:02.734 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:02.733+0000 7f198c2e3180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T01:23:02.736 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:23:02.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-19T01:23:02.789 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:23:02.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-19T01:23:02.800 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:02.798+0000 7fc8b1fa5180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T01:23:02.845 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:23:02.846 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-19T01:23:02.859 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:23:02.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-19T01:23:02.911 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:23:02.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-19T01:23:02.963 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-19T01:23:02.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-19T01:23:03.002 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:03.001+0000 7f198c2e3180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T01:23:03.018 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 48ed6785-8fb3-4d80-9062-9fc142de0020 0 2024-03-19T01:23:03.182 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb33d695640 1 Processor -- start 2024-03-19T01:23:03.183 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb33d695640 1 -- start start 2024-03-19T01:23:03.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb33d695640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb3380b5780 0x7fb33806e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb33d695640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb33809f010 0x7fb33814faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb33d695640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3380b5b60 con 0x7fb33809eac0 2024-03-19T01:23:03.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb33d695640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fb33806d9d0 con 0x7fb3380b5780 2024-03-19T01:23:03.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb33d695640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb33806d6f0 con 0x7fb33809f010 2024-03-19T01:23:03.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb3377fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb33809f010 0x7fb33814faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb3377fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb33809f010 0x7fb33814faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:33804/0 (socket says 172.21.15.167:33804) 2024-03-19T01:23:03.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb3377fe640 1 -- 172.21.15.167:0/2911485809 learned_addr learned my addr 172.21.15.167:0/2911485809 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:03.185 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb336ffd640 1 --2- 172.21.15.167:0/2911485809 >> v2:172.21.15.7:3300/0 conn(0x7fb3380b5780 0x7fb33806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.185 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.178+0000 7fb3377fe640 1 -- 172.21.15.167:0/2911485809 >> v2:172.21.15.7:3300/0 conn(0x7fb3380b5780 msgr2=0x7fb33806e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:03.185 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb3377fe640 1 --2- 172.21.15.167:0/2911485809 >> v2:172.21.15.7:3300/0 conn(0x7fb3380b5780 0x7fb33806e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.185 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb3377fe640 1 -- 172.21.15.167:0/2911485809 >> v1:172.21.15.7:6789/0 conn(0x7fb33809eac0 legacy=0x7fb3380726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:03.185 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb3377fe640 1 -- 172.21.15.167:0/2911485809 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33806def0 con 0x7fb33809f010 2024-03-19T01:23:03.186 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb3377fe640 1 --2- 172.21.15.167:0/2911485809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb33809f010 0x7fb33814faf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb3380b72c0 tx=0x7fb32801ce20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbf99e9a52545a4a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:03.186 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb335ffb640 1 -- 172.21.15.167:0/2911485809 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3280b0070 con 0x7fb33809f010 2024-03-19T01:23:03.186 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb335ffb640 1 -- 172.21.15.167:0/2911485809 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3280ab600 con 0x7fb33809f010 2024-03-19T01:23:03.186 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb335ffb640 1 -- 172.21.15.167:0/2911485809 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3280ab9d0 con 0x7fb33809f010 2024-03-19T01:23:03.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb33d695640 1 -- 172.21.15.167:0/2911485809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb33809f010 msgr2=0x7fb33814faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb33d695640 1 --2- 172.21.15.167:0/2911485809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb33809f010 0x7fb33814faf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb3380b72c0 tx=0x7fb32801ce20 comp rx=0 tx=0).stop 2024-03-19T01:23:03.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb33d695640 1 -- 172.21.15.167:0/2911485809 shutdown_connections 2024-03-19T01:23:03.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb33d695640 1 --2- 172.21.15.167:0/2911485809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb33809f010 0x7fb33814faf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb33d695640 1 --2- 172.21.15.167:0/2911485809 >> v2:172.21.15.7:3300/0 conn(0x7fb3380b5780 0x7fb33806e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb33d695640 1 -- 172.21.15.167:0/2911485809 >> 172.21.15.167:0/2911485809 conn(0x7fb3380b6610 msgr2=0x7fb3380b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:03.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb33d695640 1 -- 172.21.15.167:0/2911485809 shutdown_connections 2024-03-19T01:23:03.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.182+0000 7fb33d695640 1 -- 172.21.15.167:0/2911485809 wait complete. 2024-03-19T01:23:03.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 Processor -- start 2024-03-19T01:23:03.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 -- start start 2024-03-19T01:23:03.190 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33809eac0 0x7fb33813e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.190 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb33809f010 0x7fb3381e1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.190 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3380b5780 0x7fb3381e16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.190 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381488e0 con 0x7fb33809eac0 2024-03-19T01:23:03.190 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb338148760 con 0x7fb33809f010 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb338152850 con 0x7fb3380b5780 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3377fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3380b5780 0x7fb3381e16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3377fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3380b5780 0x7fb3381e16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:33820/0 (socket says 172.21.15.167:33820) 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3377fe640 1 -- 172.21.15.167:0/423564226 learned_addr learned my addr 172.21.15.167:0/423564226 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3367fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb33809f010 0x7fb3381e1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb336ffd640 1 --2- 172.21.15.167:0/423564226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33809eac0 0x7fb33813e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3377fe640 1 -- 172.21.15.167:0/423564226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb33809f010 msgr2=0x7fb3381e1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3377fe640 1 --2- 172.21.15.167:0/423564226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb33809f010 0x7fb3381e1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3377fe640 1 -- 172.21.15.167:0/423564226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33809eac0 msgr2=0x7fb33813e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3377fe640 1 --2- 172.21.15.167:0/423564226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33809eac0 0x7fb33813e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3377fe640 1 -- 172.21.15.167:0/423564226 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb338140cd0 con 0x7fb3380b5780 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb3377fe640 1 --2- 172.21.15.167:0/423564226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3380b5780 0x7fb3381e16c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb3280ae040 tx=0x7fb328019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb32f7fe640 1 -- 172.21.15.167:0/423564226 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3280b0040 con 0x7fb3380b5780 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb32f7fe640 1 -- 172.21.15.167:0/423564226 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3280b9070 con 0x7fb3380b5780 2024-03-19T01:23:03.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 -- 172.21.15.167:0/423564226 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb33813f630 con 0x7fb3380b5780 2024-03-19T01:23:03.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb32f7fe640 1 -- 172.21.15.167:0/423564226 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3280b4670 con 0x7fb3380b5780 2024-03-19T01:23:03.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 -- 172.21.15.167:0/423564226 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3380b36a0 con 0x7fb3380b5780 2024-03-19T01:23:03.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb32f7fe640 1 -- 172.21.15.167:0/423564226 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb3280b4810 con 0x7fb3380b5780 2024-03-19T01:23:03.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb32f7fe640 1 -- 172.21.15.167:0/423564226 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fb3280be9a0 con 0x7fb3380b5780 2024-03-19T01:23:03.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.186+0000 7fb33d695640 1 -- 172.21.15.167:0/423564226 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3380b2930 con 0x7fb3380b5780 2024-03-19T01:23:03.198 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.194+0000 7fb32f7fe640 1 -- 172.21.15.167:0/423564226 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb3280160d0 con 0x7fb3380b5780 2024-03-19T01:23:03.266 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.262+0000 7fb33d695640 1 -- 172.21.15.167:0/423564226 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd new", "uuid": "48ed6785-8fb3-4d80-9062-9fc142de0020", "id": 0} v 0) -- 0x7fb338068630 con 0x7fb3380b5780 2024-03-19T01:23:03.272 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb32f7fe640 1 -- 172.21.15.167:0/423564226 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "48ed6785-8fb3-4d80-9062-9fc142de0020", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fb3280be460 con 0x7fb3380b5780 2024-03-19T01:23:03.272 INFO:teuthology.orchestra.run.smithi167.stdout:0 2024-03-19T01:23:03.274 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb33d695640 1 -- 172.21.15.167:0/423564226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3380b5780 msgr2=0x7fb3381e16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.274 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb33d695640 1 --2- 172.21.15.167:0/423564226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3380b5780 0x7fb3381e16c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb3280ae040 tx=0x7fb328019160 comp rx=0 tx=0).stop 2024-03-19T01:23:03.274 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb33d695640 1 -- 172.21.15.167:0/423564226 shutdown_connections 2024-03-19T01:23:03.274 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb33d695640 1 --2- 172.21.15.167:0/423564226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3380b5780 0x7fb3381e16c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.274 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb33d695640 1 --2- 172.21.15.167:0/423564226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb33809f010 0x7fb3381e1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.274 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb33d695640 1 --2- 172.21.15.167:0/423564226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33809eac0 0x7fb33813e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.274 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb33d695640 1 -- 172.21.15.167:0/423564226 >> 172.21.15.167:0/423564226 conn(0x7fb3380b6610 msgr2=0x7fb3380b4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:03.274 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb33d695640 1 -- 172.21.15.167:0/423564226 shutdown_connections 2024-03-19T01:23:03.274 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.270+0000 7fb33d695640 1 -- 172.21.15.167:0/423564226 wait complete. 2024-03-19T01:23:03.278 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:03.274+0000 7fc8b1fa5180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T01:23:03.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new e4afb6c3-9411-49f9-9eb5-2d3468ef10b6 1 2024-03-19T01:23:03.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb893bb7640 1 Processor -- start 2024-03-19T01:23:03.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb893bb7640 1 -- start start 2024-03-19T01:23:03.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb893bb7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb88c0b01c0 0x7fb88c0b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb893bb7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb88c072bf0 con 0x7fb88c072810 2024-03-19T01:23:03.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb893bb7640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb88c0ba3a0 con 0x7fb88c13f090 2024-03-19T01:23:03.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb893bb7640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fb88c0afa70 con 0x7fb88c0b01c0 2024-03-19T01:23:03.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb892154640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fb88c13f090 0x7fb88c14fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:45674/0 (socket says 172.21.15.167:45674) 2024-03-19T01:23:03.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb892154640 1 -- 172.21.15.167:0/2191840989 learned_addr learned my addr 172.21.15.167:0/2191840989 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-19T01:23:03.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb891152640 1 --2- 172.21.15.167:0/2191840989 >> v2:172.21.15.7:3300/0 conn(0x7fb88c0b01c0 0x7fb88c0b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.394+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2363411010 0 0) 0x7fb88c0ba3a0 con 0x7fb88c13f090 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb878003530 con 0x7fb88c13f090 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2451661632 0 0) 0x7fb878003530 con 0x7fb88c13f090 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 >> v2:172.21.15.7:3300/0 conn(0x7fb88c0b01c0 msgr2=0x7fb88c0b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 --2- 172.21.15.167:0/2191840989 >> v2:172.21.15.7:3300/0 conn(0x7fb88c0b01c0 0x7fb88c0b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 >> v1:172.21.15.7:6789/0 conn(0x7fb88c072810 legacy=0x7fb88c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88c0ba680 con 0x7fb88c13f090 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fb884001a00 con 0x7fb88c13f090 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 >> v1:172.21.15.167:6789/0 conn(0x7fb88c13f090 legacy=0x7fb88c14fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8780039c0 0x7fb878003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb878004810 0x7fb878004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8780051d0 0x7fb87800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb87800c040 con 0x7fb8780051d0 2024-03-19T01:23:03.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb87800c1f0 con 0x7fb878004810 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb87800c3f0 con 0x7fb8780039c0 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb891953640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8780039c0 0x7fb878003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb891152640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb878004810 0x7fb878004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb892154640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8780051d0 0x7fb87800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb891953640 1 -- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb878004810 msgr2=0x7fb878004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb891953640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb878004810 0x7fb878004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb891953640 1 -- 172.21.15.167:0/2191840989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8780051d0 msgr2=0x7fb87800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb891953640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8780051d0 0x7fb87800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb892154640 1 -- 172.21.15.167:0/2191840989 reap_dead start 2024-03-19T01:23:03.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb891953640 1 -- 172.21.15.167:0/2191840989 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb878003530 con 0x7fb8780039c0 2024-03-19T01:23:03.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb891953640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8780039c0 0x7fb878003de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb888001780 tx=0x7fb88801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72520a1186df9b11 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:03.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880013a0 con 0x7fb8780039c0 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8880052f0 con 0x7fb8780039c0 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb890951640 1 -- 172.21.15.167:0/2191840989 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888003840 con 0x7fb8780039c0 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb893bb7640 1 -- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8780039c0 msgr2=0x7fb878003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb893bb7640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8780039c0 0x7fb878003de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb888001780 tx=0x7fb88801c400 comp rx=0 tx=0).stop 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb893bb7640 1 -- 172.21.15.167:0/2191840989 shutdown_connections 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb893bb7640 1 --2- 172.21.15.167:0/2191840989 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8780039c0 0x7fb878003de0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb893bb7640 1 -- 172.21.15.167:0/2191840989 >> 172.21.15.167:0/2191840989 conn(0x7fb88c06e1b0 msgr2=0x7fb88c06e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb893bb7640 1 -- 172.21.15.167:0/2191840989 shutdown_connections 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.398+0000 7fb893bb7640 1 -- 172.21.15.167:0/2191840989 wait complete. 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 Processor -- start 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 -- start start 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8780039c0 0x7fb88c1d9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb88c0b2a70 0x7fb88c1dbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb88c0b2ef0 0x7fb88c1dc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c13f8d0 con 0x7fb88c0b2a70 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c13f750 con 0x7fb8780039c0 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb88c13fa50 con 0x7fb88c0b2ef0 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891953640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8780039c0 0x7fb88c1d9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891953640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8780039c0 0x7fb88c1d9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:56674/0 (socket says 172.21.15.167:56674) 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891953640 1 -- 172.21.15.167:0/820267324 learned_addr learned my addr 172.21.15.167:0/820267324 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb892154640 1 --2- 172.21.15.167:0/820267324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb88c0b2ef0 0x7fb88c1dc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891953640 1 -- 172.21.15.167:0/820267324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb88c0b2ef0 msgr2=0x7fb88c1dc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891953640 1 --2- 172.21.15.167:0/820267324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb88c0b2ef0 0x7fb88c1dc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891152640 1 --2- 172.21.15.167:0/820267324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb88c0b2a70 0x7fb88c1dbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891953640 1 -- 172.21.15.167:0/820267324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb88c0b2a70 msgr2=0x7fb88c1dbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891953640 1 --2- 172.21.15.167:0/820267324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb88c0b2a70 0x7fb88c1dbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891953640 1 -- 172.21.15.167:0/820267324 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88c1400e0 con 0x7fb8780039c0 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb891953640 1 --2- 172.21.15.167:0/820267324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8780039c0 0x7fb88c1d9470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb88801c230 tx=0x7fb878005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb8827fc640 1 -- 172.21.15.167:0/820267324 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880b8020 con 0x7fb8780039c0 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb8827fc640 1 -- 172.21.15.167:0/820267324 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb888003e20 con 0x7fb8780039c0 2024-03-19T01:23:03.408 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 -- 172.21.15.167:0/820267324 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb88c13fe00 con 0x7fb8780039c0 2024-03-19T01:23:03.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb8827fc640 1 -- 172.21.15.167:0/820267324 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888007a40 con 0x7fb8780039c0 2024-03-19T01:23:03.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 -- 172.21.15.167:0/820267324 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb88c1d8c70 con 0x7fb8780039c0 2024-03-19T01:23:03.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb8827fc640 1 -- 172.21.15.167:0/820267324 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb8880bb020 con 0x7fb8780039c0 2024-03-19T01:23:03.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb8827fc640 1 -- 172.21.15.167:0/820267324 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fb8880b4870 con 0x7fb8780039c0 2024-03-19T01:23:03.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.402+0000 7fb893bb7640 1 -- 172.21.15.167:0/820267324 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb88c075570 con 0x7fb8780039c0 2024-03-19T01:23:03.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.406+0000 7fb8827fc640 1 -- 172.21.15.167:0/820267324 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb8880c2030 con 0x7fb8780039c0 2024-03-19T01:23:03.465 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:03.462+0000 7fc8b1fa5180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T01:23:03.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.474+0000 7fb893bb7640 1 -- 172.21.15.167:0/820267324 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e4afb6c3-9411-49f9-9eb5-2d3468ef10b6", "id": 1} v 0) -- 0x7fb88c13ded0 con 0x7fb8780039c0 2024-03-19T01:23:03.481 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb8827fc640 1 -- 172.21.15.167:0/820267324 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e4afb6c3-9411-49f9-9eb5-2d3468ef10b6", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fb88c1400e0 con 0x7fb8780039c0 2024-03-19T01:23:03.481 INFO:teuthology.orchestra.run.smithi167.stdout:1 2024-03-19T01:23:03.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb893bb7640 1 -- 172.21.15.167:0/820267324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8780039c0 msgr2=0x7fb88c1d9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb893bb7640 1 --2- 172.21.15.167:0/820267324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8780039c0 0x7fb88c1d9470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb88801c230 tx=0x7fb878005840 comp rx=0 tx=0).stop 2024-03-19T01:23:03.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb893bb7640 1 -- 172.21.15.167:0/820267324 shutdown_connections 2024-03-19T01:23:03.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb893bb7640 1 --2- 172.21.15.167:0/820267324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb88c0b2ef0 0x7fb88c1dc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb893bb7640 1 --2- 172.21.15.167:0/820267324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb88c0b2a70 0x7fb88c1dbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb893bb7640 1 --2- 172.21.15.167:0/820267324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8780039c0 0x7fb88c1d9470 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb893bb7640 1 -- 172.21.15.167:0/820267324 >> 172.21.15.167:0/820267324 conn(0x7fb88c06e1b0 msgr2=0x7fb88c06ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:03.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb893bb7640 1 -- 172.21.15.167:0/820267324 shutdown_connections 2024-03-19T01:23:03.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.478+0000 7fb893bb7640 1 -- 172.21.15.167:0/820267324 wait complete. 2024-03-19T01:23:03.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 33aff43c-f5ce-41ab-89b0-26abd458983c 2 2024-03-19T01:23:03.615 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.610+0000 7ff9361d7640 1 Processor -- start 2024-03-19T01:23:03.615 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.610+0000 7ff9361d7640 1 -- start start 2024-03-19T01:23:03.616 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff930074390 0x7ff93014abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.616 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff930074770 con 0x7ff930141430 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9300af0b0 con 0x7ff9300ae070 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7ff93014b130 con 0x7ff930074390 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92f7fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff930074390 0x7ff93014abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92f7fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff930074390 0x7ff93014abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:56682/0 (socket says 172.21.15.167:56682) 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92f7fe640 1 -- 172.21.15.167:0/3994658649 learned_addr learned my addr 172.21.15.167:0/3994658649 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92effd640 1 --1- 172.21.15.167:0/3994658649 >> v1:172.21.15.167:6789/0 conn(0x7ff9300ae070 0x7ff9300ae750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:45678/0 (socket says 172.21.15.167:45678) 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92f7fe640 1 -- 172.21.15.167:0/3994658649 >> v1:172.21.15.167:6789/0 conn(0x7ff9300ae070 legacy=0x7ff9300ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92f7fe640 1 -- 172.21.15.167:0/3994658649 >> v1:172.21.15.7:6789/0 conn(0x7ff930141430 legacy=0x7ff9301418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92e7fc640 1 -- 172.21.15.167:0/3994658649 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2509612022 0 0) 0x7ff9300af0b0 con 0x7ff9300ae070 2024-03-19T01:23:03.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92f7fe640 1 -- 172.21.15.167:0/3994658649 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9300ad870 con 0x7ff930074390 2024-03-19T01:23:03.619 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92f7fe640 1 --2- 172.21.15.167:0/3994658649 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 0x7ff93014abf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff920009870 tx=0x7ff92001c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96112e6fabbd21da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:03.619 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92e7fc640 1 -- 172.21.15.167:0/3994658649 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200b0070 con 0x7ff930074390 2024-03-19T01:23:03.619 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92e7fc640 1 -- 172.21.15.167:0/3994658649 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9200ab600 con 0x7ff930074390 2024-03-19T01:23:03.619 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff92e7fc640 1 -- 172.21.15.167:0/3994658649 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200ab9d0 con 0x7ff930074390 2024-03-19T01:23:03.620 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 -- 172.21.15.167:0/3994658649 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 msgr2=0x7ff93014abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.620 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 --2- 172.21.15.167:0/3994658649 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 0x7ff93014abf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff920009870 tx=0x7ff92001c520 comp rx=0 tx=0).stop 2024-03-19T01:23:03.620 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 -- 172.21.15.167:0/3994658649 shutdown_connections 2024-03-19T01:23:03.620 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 --2- 172.21.15.167:0/3994658649 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 0x7ff93014abf0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.620 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 -- 172.21.15.167:0/3994658649 >> 172.21.15.167:0/3994658649 conn(0x7ff93006bc70 msgr2=0x7ff9300b9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:03.620 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 -- 172.21.15.167:0/3994658649 shutdown_connections 2024-03-19T01:23:03.620 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.614+0000 7ff9361d7640 1 -- 172.21.15.167:0/3994658649 wait complete. 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 Processor -- start 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 -- start start 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 0x7ff930147a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff9300ae070 0x7ff93013ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff930141430 0x7ff93013b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff930155ae0 con 0x7ff930141430 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff930155960 con 0x7ff930074390 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff930155c60 con 0x7ff9300ae070 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92f7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 0x7ff930147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92f7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 0x7ff930147a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:56690/0 (socket says 172.21.15.167:56690) 2024-03-19T01:23:03.623 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92f7fe640 1 -- 172.21.15.167:0/2106183089 learned_addr learned my addr 172.21.15.167:0/2106183089 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92effd640 1 --2- 172.21.15.167:0/2106183089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff9300ae070 0x7ff93013ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92f7fe640 1 -- 172.21.15.167:0/2106183089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff9300ae070 msgr2=0x7ff93013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92f7fe640 1 --2- 172.21.15.167:0/2106183089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff9300ae070 0x7ff93013ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92f7fe640 1 -- 172.21.15.167:0/2106183089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff930141430 msgr2=0x7ff93013b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92f7fe640 1 --2- 172.21.15.167:0/2106183089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff930141430 0x7ff93013b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92f7fe640 1 -- 172.21.15.167:0/2106183089 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff930143420 con 0x7ff930074390 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff92f7fe640 1 --2- 172.21.15.167:0/2106183089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 0x7ff930147a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff92001abd0 tx=0x7ff9200192c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff917fff640 1 -- 172.21.15.167:0/2106183089 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200bb070 con 0x7ff930074390 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff917fff640 1 -- 172.21.15.167:0/2106183089 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9200b6630 con 0x7ff930074390 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff917fff640 1 -- 172.21.15.167:0/2106183089 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200b0040 con 0x7ff930074390 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 -- 172.21.15.167:0/2106183089 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff93013bd10 con 0x7ff930074390 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 -- 172.21.15.167:0/2106183089 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff930143140 con 0x7ff930074390 2024-03-19T01:23:03.624 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff917fff640 1 -- 172.21.15.167:0/2106183089 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff9200be070 con 0x7ff930074390 2024-03-19T01:23:03.625 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff917fff640 1 -- 172.21.15.167:0/2106183089 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7ff9200b57f0 con 0x7ff930074390 2024-03-19T01:23:03.625 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.618+0000 7ff9361d7640 1 -- 172.21.15.167:0/2106183089 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff930075570 con 0x7ff930074390 2024-03-19T01:23:03.625 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.622+0000 7ff917fff640 1 -- 172.21.15.167:0/2106183089 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff920016110 con 0x7ff930074390 2024-03-19T01:23:03.664 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:03.661+0000 7f198c2e3180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T01:23:03.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.690+0000 7ff9361d7640 1 -- 172.21.15.167:0/2106183089 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "33aff43c-f5ce-41ab-89b0-26abd458983c", "id": 2} v 0) -- 0x7ff9300b38e0 con 0x7ff930074390 2024-03-19T01:23:03.699 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.694+0000 7ff917fff640 1 -- 172.21.15.167:0/2106183089 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "33aff43c-f5ce-41ab-89b0-26abd458983c", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7ff9200ab4c0 con 0x7ff930074390 2024-03-19T01:23:03.699 INFO:teuthology.orchestra.run.smithi167.stdout:2 2024-03-19T01:23:03.700 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.698+0000 7ff9361d7640 1 -- 172.21.15.167:0/2106183089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 msgr2=0x7ff930147a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.700 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.698+0000 7ff9361d7640 1 --2- 172.21.15.167:0/2106183089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 0x7ff930147a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff92001abd0 tx=0x7ff9200192c0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.700 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.698+0000 7ff9361d7640 1 -- 172.21.15.167:0/2106183089 shutdown_connections 2024-03-19T01:23:03.700 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.698+0000 7ff9361d7640 1 --2- 172.21.15.167:0/2106183089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff9300ae070 0x7ff93013ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.698+0000 7ff9361d7640 1 --2- 172.21.15.167:0/2106183089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff930141430 0x7ff93013b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.698+0000 7ff9361d7640 1 --2- 172.21.15.167:0/2106183089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff930074390 0x7ff930147a20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.698+0000 7ff9361d7640 1 -- 172.21.15.167:0/2106183089 >> 172.21.15.167:0/2106183089 conn(0x7ff93006bc70 msgr2=0x7ff9300ae540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:03.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.698+0000 7ff9361d7640 1 -- 172.21.15.167:0/2106183089 shutdown_connections 2024-03-19T01:23:03.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.698+0000 7ff9361d7640 1 -- 172.21.15.167:0/2106183089 wait complete. 2024-03-19T01:23:03.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new c2000d0f-e3b5-43de-8b20-2494142dbc91 3 2024-03-19T01:23:03.741 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:03.737+0000 7f198c2e3180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T01:23:03.796 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:03.793+0000 7f198c2e3180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T01:23:03.827 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.822+0000 7f8fa9eea640 1 Processor -- start 2024-03-19T01:23:03.827 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.822+0000 7f8fa9eea640 1 -- start start 2024-03-19T01:23:03.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa4074390 0x7f8fa413f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f8fa4148df0 0x7f8fa414fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa4074770 con 0x7f8fa406fa70 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f8fa4070fb0 con 0x7f8fa4148df0 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa406ff90 con 0x7f8fa4074390 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa37fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa4074390 0x7f8fa413f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa37fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa4074390 0x7f8fa413f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:33860/0 (socket says 172.21.15.167:33860) 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa3fff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f8fa4148df0 0x7f8fa414fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa37fe640 1 -- 172.21.15.167:0/1701598138 learned_addr learned my addr 172.21.15.167:0/1701598138 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa2ffd640 1 --1- 172.21.15.167:0/1701598138 >> v1:172.21.15.167:6789/0 conn(0x7f8fa406fa70 0x7f8fa41486e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:45688/0 (socket says 172.21.15.167:45688) 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa37fe640 1 -- 172.21.15.167:0/1701598138 >> v2:172.21.15.167:3300/0 conn(0x7f8fa4148df0 msgr2=0x7f8fa414fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa37fe640 1 --2- 172.21.15.167:0/1701598138 >> v2:172.21.15.167:3300/0 conn(0x7f8fa4148df0 0x7f8fa414fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa37fe640 1 -- 172.21.15.167:0/1701598138 >> v1:172.21.15.167:6789/0 conn(0x7f8fa406fa70 legacy=0x7f8fa41486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa37fe640 1 -- 172.21.15.167:0/1701598138 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa4070cd0 con 0x7f8fa4074390 2024-03-19T01:23:03.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa37fe640 1 --2- 172.21.15.167:0/1701598138 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa4074390 0x7f8fa413f1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009e50 tx=0x7f8f9401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef0daaeb186d7046 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:03.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa27fc640 1 -- 172.21.15.167:0/1701598138 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f940a5810 con 0x7f8fa4074390 2024-03-19T01:23:03.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa27fc640 1 -- 172.21.15.167:0/1701598138 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f940a59b0 con 0x7f8fa4074390 2024-03-19T01:23:03.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa27fc640 1 -- 172.21.15.167:0/1701598138 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f940a5d80 con 0x7f8fa4074390 2024-03-19T01:23:03.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1701598138 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa4074390 msgr2=0x7f8fa413f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 --2- 172.21.15.167:0/1701598138 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa4074390 0x7f8fa413f1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009e50 tx=0x7f8f9401ca30 comp rx=0 tx=0).stop 2024-03-19T01:23:03.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1701598138 shutdown_connections 2024-03-19T01:23:03.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 --2- 172.21.15.167:0/1701598138 >> v2:172.21.15.167:3300/0 conn(0x7f8fa4148df0 0x7f8fa414fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 --2- 172.21.15.167:0/1701598138 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa4074390 0x7f8fa413f1c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1701598138 >> 172.21.15.167:0/1701598138 conn(0x7f8fa40b5c90 msgr2=0x7f8fa40b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:03.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1701598138 shutdown_connections 2024-03-19T01:23:03.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.826+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1701598138 wait complete. 2024-03-19T01:23:03.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 Processor -- start 2024-03-19T01:23:03.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 -- start start 2024-03-19T01:23:03.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa406fa70 0x7f8fa41e17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fa4074390 0x7f8fa4143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fa4148df0 0x7f8fa4145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa413f880 con 0x7f8fa4148df0 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa413f700 con 0x7f8fa4074390 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa4152820 con 0x7f8fa406fa70 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa37fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa406fa70 0x7f8fa41e17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa37fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa406fa70 0x7f8fa41e17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:33864/0 (socket says 172.21.15.167:33864) 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa37fe640 1 -- 172.21.15.167:0/1804484798 learned_addr learned my addr 172.21.15.167:0/1804484798 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa2ffd640 1 --2- 172.21.15.167:0/1804484798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fa4074390 0x7f8fa4143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa3fff640 1 --2- 172.21.15.167:0/1804484798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fa4148df0 0x7f8fa4145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa37fe640 1 -- 172.21.15.167:0/1804484798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fa4074390 msgr2=0x7f8fa4143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa37fe640 1 --2- 172.21.15.167:0/1804484798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fa4074390 0x7f8fa4143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa37fe640 1 -- 172.21.15.167:0/1804484798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fa4148df0 msgr2=0x7f8fa4145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa37fe640 1 --2- 172.21.15.167:0/1804484798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fa4148df0 0x7f8fa4145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa37fe640 1 -- 172.21.15.167:0/1804484798 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa41e1df0 con 0x7f8fa406fa70 2024-03-19T01:23:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa37fe640 1 --2- 172.21.15.167:0/1804484798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa406fa70 0x7f8fa41e17c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8f9401a660 tx=0x7f8f940197d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8f9bfff640 1 -- 172.21.15.167:0/1804484798 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f940bc020 con 0x7f8fa406fa70 2024-03-19T01:23:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8f9bfff640 1 -- 172.21.15.167:0/1804484798 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f940b8670 con 0x7f8fa406fa70 2024-03-19T01:23:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8f9bfff640 1 -- 172.21.15.167:0/1804484798 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f940b8a40 con 0x7f8fa406fa70 2024-03-19T01:23:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1804484798 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa41e20d0 con 0x7f8fa406fa70 2024-03-19T01:23:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1804484798 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa4143770 con 0x7f8fa406fa70 2024-03-19T01:23:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8f9bfff640 1 -- 172.21.15.167:0/1804484798 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8f940bf020 con 0x7f8fa406fa70 2024-03-19T01:23:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1804484798 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa4075570 con 0x7f8fa406fa70 2024-03-19T01:23:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.830+0000 7f8f9bfff640 1 -- 172.21.15.167:0/1804484798 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f8f94004690 con 0x7f8fa406fa70 2024-03-19T01:23:03.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.834+0000 7f8f9bfff640 1 -- 172.21.15.167:0/1804484798 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8f940c7020 con 0x7f8fa406fa70 2024-03-19T01:23:03.878 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:03.877+0000 7f198c2e3180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T01:23:03.904 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.902+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1804484798 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c2000d0f-e3b5-43de-8b20-2494142dbc91", "id": 3} v 0) -- 0x7f8fa414fc60 con 0x7f8fa406fa70 2024-03-19T01:23:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.906+0000 7f8f9bfff640 1 -- 172.21.15.167:0/1804484798 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c2000d0f-e3b5-43de-8b20-2494142dbc91", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f8f94016620 con 0x7f8fa406fa70 2024-03-19T01:23:03.911 INFO:teuthology.orchestra.run.smithi167.stdout:3 2024-03-19T01:23:03.912 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.910+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1804484798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa406fa70 msgr2=0x7f8fa41e17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:03.912 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.910+0000 7f8fa9eea640 1 --2- 172.21.15.167:0/1804484798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa406fa70 0x7f8fa41e17c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8f9401a660 tx=0x7f8f940197d0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.910+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1804484798 shutdown_connections 2024-03-19T01:23:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.910+0000 7f8fa9eea640 1 --2- 172.21.15.167:0/1804484798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fa4148df0 0x7f8fa4145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.910+0000 7f8fa9eea640 1 --2- 172.21.15.167:0/1804484798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fa4074390 0x7f8fa4143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.910+0000 7f8fa9eea640 1 --2- 172.21.15.167:0/1804484798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fa406fa70 0x7f8fa41e17c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.910+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1804484798 >> 172.21.15.167:0/1804484798 conn(0x7f8fa40b5c90 msgr2=0x7f8fa40b7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.910+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1804484798 shutdown_connections 2024-03-19T01:23:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:03.910+0000 7f8fa9eea640 1 -- 172.21.15.167:0/1804484798 wait complete. 2024-03-19T01:23:03.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new eba08682-fcb1-48ae-bd37-03af0cac45d2 4 2024-03-19T01:23:03.932 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:03.929+0000 7f198c2e3180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T01:23:04.013 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:04.009+0000 7f198c2e3180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T01:23:04.029 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:04.026+0000 7fc8b1fa5180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T01:23:04.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f65a5115640 1 Processor -- start 2024-03-19T01:23:04.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f65a5115640 1 -- start start 2024-03-19T01:23:04.040 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f65a5115640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65a00b77f0 0x7f65a00b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.040 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f65a5115640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b69f0 0x7f65a00b4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.040 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f65a5115640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a00b2c20 con 0x7f65a00b6e50 2024-03-19T01:23:04.040 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f65a5115640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f65a00b5d00 con 0x7f65a00b77f0 2024-03-19T01:23:04.040 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f65a5115640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f65a00b5fe0 con 0x7f65a00b69f0 2024-03-19T01:23:04.040 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f659e59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b69f0 0x7f65a00b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.040 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f659e59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b69f0 0x7f65a00b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:33870/0 (socket says 172.21.15.167:33870) 2024-03-19T01:23:04.041 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f659e59c640 1 -- 172.21.15.167:0/942720908 learned_addr learned my addr 172.21.15.167:0/942720908 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:04.041 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.034+0000 7f659ed9d640 1 --2- 172.21.15.167:0/942720908 >> v2:172.21.15.7:3300/0 conn(0x7f65a00b77f0 0x7f65a00b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.041 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f659e59c640 1 -- 172.21.15.167:0/942720908 >> v2:172.21.15.7:3300/0 conn(0x7f65a00b77f0 msgr2=0x7f65a00b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:04.041 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f659e59c640 1 --2- 172.21.15.167:0/942720908 >> v2:172.21.15.7:3300/0 conn(0x7f65a00b77f0 0x7f65a00b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.041 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f659e59c640 1 -- 172.21.15.167:0/942720908 >> v1:172.21.15.7:6789/0 conn(0x7f65a00b6e50 legacy=0x7f65a00714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:04.041 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f659e59c640 1 -- 172.21.15.167:0/942720908 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a00b64d0 con 0x7f65a00b69f0 2024-03-19T01:23:04.042 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f659e59c640 1 --2- 172.21.15.167:0/942720908 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b69f0 0x7f65a00b4380 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6594009870 tx=0x7f659401c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a90cbd2062926ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.042 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f659dd9b640 1 -- 172.21.15.167:0/942720908 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65940a7070 con 0x7f65a00b69f0 2024-03-19T01:23:04.042 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f659dd9b640 1 -- 172.21.15.167:0/942720908 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65940b0070 con 0x7f65a00b69f0 2024-03-19T01:23:04.042 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f659dd9b640 1 -- 172.21.15.167:0/942720908 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65940ab890 con 0x7f65a00b69f0 2024-03-19T01:23:04.043 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f65a5115640 1 -- 172.21.15.167:0/942720908 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b69f0 msgr2=0x7f65a00b4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.043 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f65a5115640 1 --2- 172.21.15.167:0/942720908 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b69f0 0x7f65a00b4380 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6594009870 tx=0x7f659401c490 comp rx=0 tx=0).stop 2024-03-19T01:23:04.043 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f65a5115640 1 -- 172.21.15.167:0/942720908 shutdown_connections 2024-03-19T01:23:04.043 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f65a5115640 1 --2- 172.21.15.167:0/942720908 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b69f0 0x7f65a00b4380 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.044 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f65a5115640 1 --2- 172.21.15.167:0/942720908 >> v2:172.21.15.7:3300/0 conn(0x7f65a00b77f0 0x7f65a00b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.044 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f65a5115640 1 -- 172.21.15.167:0/942720908 >> 172.21.15.167:0/942720908 conn(0x7f65a006b2c0 msgr2=0x7f65a006b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:04.044 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f65a5115640 1 -- 172.21.15.167:0/942720908 shutdown_connections 2024-03-19T01:23:04.044 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.038+0000 7f65a5115640 1 -- 172.21.15.167:0/942720908 wait complete. 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 Processor -- start 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 -- start start 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a00b69f0 0x7f65a013ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b6e50 0x7f65a01e35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00b77f0 0x7f65a006dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f65a01489b0 con 0x7f65a00b77f0 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f65a0148830 con 0x7f65a00b69f0 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f65a0152b80 con 0x7f65a00b6e50 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659ed9d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a00b69f0 0x7f65a013ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659e59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b6e50 0x7f65a01e35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659ed9d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a00b69f0 0x7f65a013ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:56722/0 (socket says 172.21.15.167:56722) 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659ed9d640 1 -- 172.21.15.167:0/3498006862 learned_addr learned my addr 172.21.15.167:0/3498006862 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:04.052 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659f59e640 1 --2- 172.21.15.167:0/3498006862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00b77f0 0x7f65a006dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659ed9d640 1 -- 172.21.15.167:0/3498006862 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b6e50 msgr2=0x7f65a01e35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659ed9d640 1 --2- 172.21.15.167:0/3498006862 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b6e50 0x7f65a01e35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659ed9d640 1 -- 172.21.15.167:0/3498006862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00b77f0 msgr2=0x7f65a006dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659ed9d640 1 --2- 172.21.15.167:0/3498006862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00b77f0 0x7f65a006dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659ed9d640 1 -- 172.21.15.167:0/3498006862 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a013f0e0 con 0x7f65a00b69f0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659e59c640 1 --2- 172.21.15.167:0/3498006862 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b6e50 0x7f65a01e35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f659ed9d640 1 --2- 172.21.15.167:0/3498006862 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a00b69f0 0x7f65a013ad90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f659000d5f0 tx=0x7f659000dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65877fe640 1 -- 172.21.15.167:0/3498006862 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65900125f0 con 0x7f65a00b69f0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65877fe640 1 -- 172.21.15.167:0/3498006862 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6590019070 con 0x7f65a00b69f0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 -- 172.21.15.167:0/3498006862 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a014e5d0 con 0x7f65a00b69f0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65877fe640 1 -- 172.21.15.167:0/3498006862 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65900129d0 con 0x7f65a00b69f0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 -- 172.21.15.167:0/3498006862 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a013b970 con 0x7f65a00b69f0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65877fe640 1 -- 172.21.15.167:0/3498006862 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6590012b70 con 0x7f65a00b69f0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65877fe640 1 -- 172.21.15.167:0/3498006862 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f65900149e0 con 0x7f65a00b69f0 2024-03-19T01:23:04.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.042+0000 7f65a5115640 1 -- 172.21.15.167:0/3498006862 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a0075570 con 0x7f65a00b69f0 2024-03-19T01:23:04.054 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.046+0000 7f65877fe640 1 -- 172.21.15.167:0/3498006862 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6590014cc0 con 0x7f65a00b69f0 2024-03-19T01:23:04.069 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:04.065+0000 7f198c2e3180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T01:23:04.085 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:04.082+0000 7fc8b1fa5180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T01:23:04.116 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.110+0000 7f65a5115640 1 -- 172.21.15.167:0/3498006862 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "eba08682-fcb1-48ae-bd37-03af0cac45d2", "id": 4} v 0) -- 0x7f65a014e280 con 0x7f65a00b69f0 2024-03-19T01:23:04.121 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65877fe640 1 -- 172.21.15.167:0/3498006862 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eba08682-fcb1-48ae-bd37-03af0cac45d2", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f65900144d0 con 0x7f65a00b69f0 2024-03-19T01:23:04.121 INFO:teuthology.orchestra.run.smithi167.stdout:4 2024-03-19T01:23:04.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65a5115640 1 -- 172.21.15.167:0/3498006862 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a00b69f0 msgr2=0x7f65a013ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65a5115640 1 --2- 172.21.15.167:0/3498006862 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a00b69f0 0x7f65a013ad90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f659000d5f0 tx=0x7f659000dab0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65a5115640 1 -- 172.21.15.167:0/3498006862 shutdown_connections 2024-03-19T01:23:04.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65a5115640 1 --2- 172.21.15.167:0/3498006862 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a00b6e50 0x7f65a01e35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65a5115640 1 --2- 172.21.15.167:0/3498006862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00b77f0 0x7f65a006dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65a5115640 1 --2- 172.21.15.167:0/3498006862 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a00b69f0 0x7f65a013ad90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65a5115640 1 -- 172.21.15.167:0/3498006862 >> 172.21.15.167:0/3498006862 conn(0x7f65a006b2c0 msgr2=0x7f65a0146bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:04.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65a5115640 1 -- 172.21.15.167:0/3498006862 shutdown_connections 2024-03-19T01:23:04.123 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.118+0000 7f65a5115640 1 -- 172.21.15.167:0/3498006862 wait complete. 2024-03-19T01:23:04.137 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new d76735f6-3a76-4f1f-8817-92c4c6532a98 5 2024-03-19T01:23:04.139 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:04.134+0000 7fc8b1fa5180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T01:23:04.218 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:04.214+0000 7fc8b1fa5180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T01:23:04.226 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:04.225+0000 7f198c2e3180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T01:23:04.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.250+0000 7faed2586640 1 Processor -- start 2024-03-19T01:23:04.256 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.250+0000 7faed2586640 1 -- start start 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faed2586640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc0b6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faed2586640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7faecc145b30 0x7faecc0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faed2586640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faecc0b56e0 con 0x7faecc145f90 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faed2586640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7faecc1456a0 con 0x7faecc145b30 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faed2586640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faecc0b2cb0 con 0x7faecc0b6960 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecbfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc0b6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecb7fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7faecc145b30 0x7faecc0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecbfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc0b6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:33886/0 (socket says 172.21.15.167:33886) 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecb7fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7faecc145b30 0x7faecc0b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:56734/0 (socket says 172.21.15.167:56734) 2024-03-19T01:23:04.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecbfff640 1 -- 172.21.15.167:0/2098162581 learned_addr learned my addr 172.21.15.167:0/2098162581 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:04.258 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecb7fe640 1 -- 172.21.15.167:0/2098162581 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 msgr2=0x7faecc0b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:04.258 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecb7fe640 1 --2- 172.21.15.167:0/2098162581 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc0b6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.258 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecb7fe640 1 -- 172.21.15.167:0/2098162581 >> v1:172.21.15.7:6789/0 conn(0x7faecc145f90 legacy=0x7faecc148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:04.258 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecb7fe640 1 -- 172.21.15.167:0/2098162581 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faecc0b59c0 con 0x7faecc145b30 2024-03-19T01:23:04.258 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecbfff640 1 --2- 172.21.15.167:0/2098162581 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc0b6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:04.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecb7fe640 1 --2- 172.21.15.167:0/2098162581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 0x7faecc0b44a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faec00097c0 tx=0x7faec001c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9537d912b68056 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecaffd640 1 -- 172.21.15.167:0/2098162581 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec00b1070 con 0x7faecc145b30 2024-03-19T01:23:04.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecaffd640 1 -- 172.21.15.167:0/2098162581 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec00ac410 con 0x7faecc145b30 2024-03-19T01:23:04.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faecaffd640 1 -- 172.21.15.167:0/2098162581 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec00ac7e0 con 0x7faecc145b30 2024-03-19T01:23:04.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.254+0000 7faed2586640 1 -- 172.21.15.167:0/2098162581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 msgr2=0x7faecc0b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 --2- 172.21.15.167:0/2098162581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 0x7faecc0b44a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faec00097c0 tx=0x7faec001c3d0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- 172.21.15.167:0/2098162581 shutdown_connections 2024-03-19T01:23:04.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 --2- 172.21.15.167:0/2098162581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 0x7faecc0b44a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 --2- 172.21.15.167:0/2098162581 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc0b6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- 172.21.15.167:0/2098162581 >> 172.21.15.167:0/2098162581 conn(0x7faecc0bace0 msgr2=0x7faecc0bb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:04.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- 172.21.15.167:0/2098162581 shutdown_connections 2024-03-19T01:23:04.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- 172.21.15.167:0/2098162581 wait complete. 2024-03-19T01:23:04.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 Processor -- start 2024-03-19T01:23:04.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- start start 2024-03-19T01:23:04.263 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.263 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 0x7faecc14f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.263 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc145f90 0x7faecc14fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7faecc0bbae0 con 0x7faecc145f90 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faecc0bb960 con 0x7faecc145b30 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faecc152b80 con 0x7faecc0b6960 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecbfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecb7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 0x7faecc14f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecbfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc06da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:33894/0 (socket says 172.21.15.167:33894) 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecbfff640 1 -- 172.21.15.167:0/3746994435 learned_addr learned my addr 172.21.15.167:0/3746994435 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed0b23640 1 --2- 172.21.15.167:0/3746994435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc145f90 0x7faecc14fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecbfff640 1 -- 172.21.15.167:0/3746994435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 msgr2=0x7faecc14f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecbfff640 1 --2- 172.21.15.167:0/3746994435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 0x7faecc14f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecbfff640 1 -- 172.21.15.167:0/3746994435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc145f90 msgr2=0x7faecc14fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecbfff640 1 --2- 172.21.15.167:0/3746994435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc145f90 0x7faecc14fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecbfff640 1 -- 172.21.15.167:0/3746994435 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faecc14eb30 con 0x7faecc0b6960 2024-03-19T01:23:04.264 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecb7fe640 1 --2- 172.21.15.167:0/3746994435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 0x7faecc14f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faecbfff640 1 --2- 172.21.15.167:0/3746994435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc06da50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faec4009e80 tx=0x7faec400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faec8ff9640 1 -- 172.21.15.167:0/3746994435 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec400ce60 con 0x7faecc0b6960 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faec8ff9640 1 -- 172.21.15.167:0/3746994435 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec4019070 con 0x7faecc0b6960 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- 172.21.15.167:0/3746994435 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faecc06e180 con 0x7faecc0b6960 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faec8ff9640 1 -- 172.21.15.167:0/3746994435 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec4014680 con 0x7faecc0b6960 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- 172.21.15.167:0/3746994435 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faecc14dc50 con 0x7faecc0b6960 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faec8ff9640 1 -- 172.21.15.167:0/3746994435 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faec4012050 con 0x7faecc0b6960 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faec8ff9640 1 -- 172.21.15.167:0/3746994435 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7faec401daf0 con 0x7faecc0b6960 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.258+0000 7faed2586640 1 -- 172.21.15.167:0/3746994435 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faecc075570 con 0x7faecc0b6960 2024-03-19T01:23:04.265 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.262+0000 7faec8ff9640 1 -- 172.21.15.167:0/3746994435 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7faec4010040 con 0x7faecc0b6960 2024-03-19T01:23:04.273 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:04.270+0000 7fc8b1fa5180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T01:23:04.290 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:04.289+0000 7f198c2e3180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T01:23:04.334 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.330+0000 7faed2586640 1 -- 172.21.15.167:0/3746994435 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d76735f6-3a76-4f1f-8817-92c4c6532a98", "id": 5} v 0) -- 0x7faecc06e350 con 0x7faecc0b6960 2024-03-19T01:23:04.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.334+0000 7faec8ff9640 1 -- 172.21.15.167:0/3746994435 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d76735f6-3a76-4f1f-8817-92c4c6532a98", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7faec4014d40 con 0x7faecc0b6960 2024-03-19T01:23:04.339 INFO:teuthology.orchestra.run.smithi167.stdout:5 2024-03-19T01:23:04.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.338+0000 7faed2586640 1 -- 172.21.15.167:0/3746994435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 msgr2=0x7faecc06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.338+0000 7faed2586640 1 --2- 172.21.15.167:0/3746994435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc06da50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faec4009e80 tx=0x7faec400c490 comp rx=0 tx=0).stop 2024-03-19T01:23:04.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.338+0000 7faed2586640 1 -- 172.21.15.167:0/3746994435 shutdown_connections 2024-03-19T01:23:04.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.338+0000 7faed2586640 1 --2- 172.21.15.167:0/3746994435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc145f90 0x7faecc14fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.338+0000 7faed2586640 1 --2- 172.21.15.167:0/3746994435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faecc145b30 0x7faecc14f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.338+0000 7faed2586640 1 --2- 172.21.15.167:0/3746994435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faecc0b6960 0x7faecc06da50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.338+0000 7faed2586640 1 -- 172.21.15.167:0/3746994435 >> 172.21.15.167:0/3746994435 conn(0x7faecc0bace0 msgr2=0x7faecc144760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:04.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.338+0000 7faed2586640 1 -- 172.21.15.167:0/3746994435 shutdown_connections 2024-03-19T01:23:04.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.338+0000 7faed2586640 1 -- 172.21.15.167:0/3746994435 wait complete. 2024-03-19T01:23:04.360 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:04.354+0000 7fc8b1fa5180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T01:23:04.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352 6 2024-03-19T01:23:04.415 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:04.410+0000 7fc8b1fa5180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T01:23:04.474 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d8e88640 1 Processor -- start 2024-03-19T01:23:04.474 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d8e88640 1 -- start start 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d8e88640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f49d41488a0 0x7f49d4148d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d8e88640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f49d414db30 0x7f49d414ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d8e88640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d40b1ae0 con 0x7f49d40726d0 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d8e88640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f49d40b9560 con 0x7f49d414db30 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d8e88640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f49d40b97b0 con 0x7f49d41488a0 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d1d9b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f49d41488a0 0x7f49d4148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d1d9b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f49d41488a0 0x7f49d4148d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:56742/0 (socket says 172.21.15.167:56742) 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d1d9b640 1 -- 172.21.15.167:0/3953725798 learned_addr learned my addr 172.21.15.167:0/3953725798 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d2d9d640 1 --2- 172.21.15.167:0/3953725798 >> v2:172.21.15.7:3300/0 conn(0x7f49d414db30 0x7f49d414ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d1d9b640 1 -- 172.21.15.167:0/3953725798 >> v2:172.21.15.7:3300/0 conn(0x7f49d414db30 msgr2=0x7f49d414ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d1d9b640 1 --2- 172.21.15.167:0/3953725798 >> v2:172.21.15.7:3300/0 conn(0x7f49d414db30 0x7f49d414ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d1d9b640 1 -- 172.21.15.167:0/3953725798 >> v1:172.21.15.167:6789/0 conn(0x7f49d40726d0 legacy=0x7f49d40b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.477 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d159a640 1 -- 172.21.15.167:0/3953725798 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799544480 0 0) 0x7f49d40b1ae0 con 0x7f49d40726d0 2024-03-19T01:23:04.477 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.470+0000 7f49d1d9b640 1 -- 172.21.15.167:0/3953725798 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d4072ab0 con 0x7f49d41488a0 2024-03-19T01:23:04.477 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d1d9b640 1 --2- 172.21.15.167:0/3953725798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d41488a0 0x7f49d4148d00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49d406da90 tx=0x7f49bc01c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de3719f8ee3b6167 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.477 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d159a640 1 -- 172.21.15.167:0/3953725798 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0b0070 con 0x7f49d41488a0 2024-03-19T01:23:04.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d159a640 1 -- 172.21.15.167:0/3953725798 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49bc0ab600 con 0x7f49d41488a0 2024-03-19T01:23:04.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d159a640 1 -- 172.21.15.167:0/3953725798 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0ab9d0 con 0x7f49d41488a0 2024-03-19T01:23:04.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 -- 172.21.15.167:0/3953725798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d41488a0 msgr2=0x7f49d4148d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 --2- 172.21.15.167:0/3953725798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d41488a0 0x7f49d4148d00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49d406da90 tx=0x7f49bc01c490 comp rx=0 tx=0).stop 2024-03-19T01:23:04.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 -- 172.21.15.167:0/3953725798 shutdown_connections 2024-03-19T01:23:04.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 --2- 172.21.15.167:0/3953725798 >> v2:172.21.15.7:3300/0 conn(0x7f49d414db30 0x7f49d414ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 --2- 172.21.15.167:0/3953725798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d41488a0 0x7f49d4148d00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 -- 172.21.15.167:0/3953725798 >> 172.21.15.167:0/3953725798 conn(0x7f49d40b7980 msgr2=0x7f49d40b7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:04.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 -- 172.21.15.167:0/3953725798 shutdown_connections 2024-03-19T01:23:04.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 -- 172.21.15.167:0/3953725798 wait complete. 2024-03-19T01:23:04.481 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 Processor -- start 2024-03-19T01:23:04.481 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.474+0000 7f49d8e88640 1 -- start start 2024-03-19T01:23:04.481 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d8e88640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49d40726d0 0x7f49d414f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d8e88640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49d41488a0 0x7f49d41e3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d8e88640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d414db30 0x7f49d41377c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d8e88640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f49d406eab0 con 0x7f49d41488a0 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d8e88640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f49d406e930 con 0x7f49d414db30 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d8e88640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f49d40b3b00 con 0x7f49d40726d0 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d2d9d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d414db30 0x7f49d41377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d259c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49d40726d0 0x7f49d414f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d2d9d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d414db30 0x7f49d41377c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:56752/0 (socket says 172.21.15.167:56752) 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d2d9d640 1 -- 172.21.15.167:0/813723874 learned_addr learned my addr 172.21.15.167:0/813723874 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d1d9b640 1 --2- 172.21.15.167:0/813723874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49d41488a0 0x7f49d41e3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d2d9d640 1 -- 172.21.15.167:0/813723874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49d40726d0 msgr2=0x7f49d414f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d2d9d640 1 --2- 172.21.15.167:0/813723874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49d40726d0 0x7f49d414f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d2d9d640 1 -- 172.21.15.167:0/813723874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49d41488a0 msgr2=0x7f49d41e3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d2d9d640 1 --2- 172.21.15.167:0/813723874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49d41488a0 0x7f49d41e3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d2d9d640 1 -- 172.21.15.167:0/813723874 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d414fd80 con 0x7f49d414db30 2024-03-19T01:23:04.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d2d9d640 1 --2- 172.21.15.167:0/813723874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d414db30 0x7f49d41377c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00cc00 tx=0x7f49cc00a5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49caffd640 1 -- 172.21.15.167:0/813723874 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc00ce10 con 0x7f49d414db30 2024-03-19T01:23:04.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49caffd640 1 -- 172.21.15.167:0/813723874 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49cc0039a0 con 0x7f49d414db30 2024-03-19T01:23:04.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49caffd640 1 -- 172.21.15.167:0/813723874 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc003d70 con 0x7f49d414db30 2024-03-19T01:23:04.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d8e88640 1 -- 172.21.15.167:0/813723874 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d41e3090 con 0x7f49d414db30 2024-03-19T01:23:04.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d8e88640 1 -- 172.21.15.167:0/813723874 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d414eb80 con 0x7f49d414db30 2024-03-19T01:23:04.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49caffd640 1 -- 172.21.15.167:0/813723874 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f49cc0042a0 con 0x7f49d414db30 2024-03-19T01:23:04.483 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49caffd640 1 -- 172.21.15.167:0/813723874 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f49cc018c20 con 0x7f49d414db30 2024-03-19T01:23:04.484 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.478+0000 7f49d8e88640 1 -- 172.21.15.167:0/813723874 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d41e3260 con 0x7f49d414db30 2024-03-19T01:23:04.484 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.482+0000 7f49caffd640 1 -- 172.21.15.167:0/813723874 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f49cc014030 con 0x7f49d414db30 2024-03-19T01:23:04.548 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.542+0000 7f49d8e88640 1 -- 172.21.15.167:0/813723874 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352", "id": 6} v 0) -- 0x7f49d4075570 con 0x7f49d414db30 2024-03-19T01:23:04.553 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49caffd640 1 -- 172.21.15.167:0/813723874 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f49cc004770 con 0x7f49d414db30 2024-03-19T01:23:04.553 INFO:teuthology.orchestra.run.smithi167.stdout:6 2024-03-19T01:23:04.554 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49d8e88640 1 -- 172.21.15.167:0/813723874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d414db30 msgr2=0x7f49d41377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.554 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49d8e88640 1 --2- 172.21.15.167:0/813723874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d414db30 0x7f49d41377c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00cc00 tx=0x7f49cc00a5c0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.554 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49d8e88640 1 -- 172.21.15.167:0/813723874 shutdown_connections 2024-03-19T01:23:04.554 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49d8e88640 1 --2- 172.21.15.167:0/813723874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d414db30 0x7f49d41377c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.554 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49d8e88640 1 --2- 172.21.15.167:0/813723874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49d41488a0 0x7f49d41e3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.554 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49d8e88640 1 --2- 172.21.15.167:0/813723874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49d40726d0 0x7f49d414f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.555 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49d8e88640 1 -- 172.21.15.167:0/813723874 >> 172.21.15.167:0/813723874 conn(0x7f49d40b7980 msgr2=0x7f49d40b8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:04.555 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49d8e88640 1 -- 172.21.15.167:0/813723874 shutdown_connections 2024-03-19T01:23:04.555 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.550+0000 7f49d8e88640 1 -- 172.21.15.167:0/813723874 wait complete. 2024-03-19T01:23:04.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 1bdb920f-d25e-4092-9d3f-e2cabe5e21f3 7 2024-03-19T01:23:04.570 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:04.566+0000 7fc8b1fa5180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T01:23:04.633 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:04.630+0000 7fc8b1fa5180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T01:23:04.675 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.670+0000 7f084fb6e640 1 Processor -- start 2024-03-19T01:23:04.675 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.670+0000 7f084fb6e640 1 -- start start 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08480ae250 0x7f08480b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f08480ae810 0x7f0848151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08480afb20 con 0x7f0848074390 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f0848074770 con 0x7f08480ae810 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f08480add60 con 0x7f08480ae250 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084d90a640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f0848074390 0x7f08481445d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:45700/0 (socket says 172.21.15.167:45700) 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084d90a640 1 -- 172.21.15.167:0/1037853754 learned_addr learned my addr 172.21.15.167:0/1037853754 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084d109640 1 --2- 172.21.15.167:0/1037853754 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08480ae250 0x7f08480b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084e10b640 1 --2- 172.21.15.167:0/1037853754 >> v2:172.21.15.7:3300/0 conn(0x7f08480ae810 0x7f0848151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084d109640 1 -- 172.21.15.167:0/1037853754 >> v2:172.21.15.7:3300/0 conn(0x7f08480ae810 msgr2=0x7f0848151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:04.677 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084d109640 1 --2- 172.21.15.167:0/1037853754 >> v2:172.21.15.7:3300/0 conn(0x7f08480ae810 0x7f0848151ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.678 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084d109640 1 -- 172.21.15.167:0/1037853754 >> v1:172.21.15.167:6789/0 conn(0x7f0848074390 legacy=0x7f08481445d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.678 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084c908640 1 -- 172.21.15.167:0/1037853754 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1245184530 0 0) 0x7f08480afb20 con 0x7f0848074390 2024-03-19T01:23:04.678 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084d109640 1 -- 172.21.15.167:0/1037853754 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08480b3180 con 0x7f08480ae250 2024-03-19T01:23:04.678 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084d109640 1 --2- 172.21.15.167:0/1037853754 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08480ae250 0x7f08480b5140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f08380097c0 tx=0x7f083801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b9a2e946069df94 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.679 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084c908640 1 -- 172.21.15.167:0/1037853754 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08380b0070 con 0x7f08480ae250 2024-03-19T01:23:04.679 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084c908640 1 -- 172.21.15.167:0/1037853754 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08380ab600 con 0x7f08480ae250 2024-03-19T01:23:04.679 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084c908640 1 -- 172.21.15.167:0/1037853754 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08380ab9d0 con 0x7f08480ae250 2024-03-19T01:23:04.680 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 -- 172.21.15.167:0/1037853754 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08480ae250 msgr2=0x7f08480b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.680 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 --2- 172.21.15.167:0/1037853754 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08480ae250 0x7f08480b5140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f08380097c0 tx=0x7f083801c410 comp rx=0 tx=0).stop 2024-03-19T01:23:04.680 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 -- 172.21.15.167:0/1037853754 shutdown_connections 2024-03-19T01:23:04.680 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 --2- 172.21.15.167:0/1037853754 >> v2:172.21.15.7:3300/0 conn(0x7f08480ae810 0x7f0848151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.680 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 --2- 172.21.15.167:0/1037853754 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08480ae250 0x7f08480b5140 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.680 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 -- 172.21.15.167:0/1037853754 >> 172.21.15.167:0/1037853754 conn(0x7f08480b5c90 msgr2=0x7f08480b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:04.680 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 -- 172.21.15.167:0/1037853754 shutdown_connections 2024-03-19T01:23:04.680 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.674+0000 7f084fb6e640 1 -- 172.21.15.167:0/1037853754 wait complete. 2024-03-19T01:23:04.682 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 Processor -- start 2024-03-19T01:23:04.682 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 -- start start 2024-03-19T01:23:04.682 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0848074390 0x7f084813ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.682 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08480ae810 0x7f0848145650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0848140ac0 0x7f0848145b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f08480b03a0 con 0x7f08480ae810 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f08480b0220 con 0x7f0848140ac0 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0848144ce0 con 0x7f0848074390 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084e10b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0848140ac0 0x7f0848145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084d90a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0848074390 0x7f084813ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084e10b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0848140ac0 0x7f0848145b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:56760/0 (socket says 172.21.15.167:56760) 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084e10b640 1 -- 172.21.15.167:0/1345272954 learned_addr learned my addr 172.21.15.167:0/1345272954 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084d109640 1 --2- 172.21.15.167:0/1345272954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08480ae810 0x7f0848145650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084d90a640 1 -- 172.21.15.167:0/1345272954 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0848140ac0 msgr2=0x7f0848145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084d90a640 1 --2- 172.21.15.167:0/1345272954 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0848140ac0 0x7f0848145b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084d90a640 1 -- 172.21.15.167:0/1345272954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08480ae810 msgr2=0x7f0848145650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084d90a640 1 --2- 172.21.15.167:0/1345272954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08480ae810 0x7f0848145650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084d90a640 1 -- 172.21.15.167:0/1345272954 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08481469a0 con 0x7f0848074390 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084e10b640 1 --2- 172.21.15.167:0/1345272954 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0848140ac0 0x7f0848145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084d90a640 1 --2- 172.21.15.167:0/1345272954 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0848074390 0x7f084813ed50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f084400c430 tx=0x7f084400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f083e7fc640 1 -- 172.21.15.167:0/1345272954 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08440074e0 con 0x7f0848074390 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f083e7fc640 1 -- 172.21.15.167:0/1345272954 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08440047f0 con 0x7f0848074390 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 -- 172.21.15.167:0/1345272954 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0848146c80 con 0x7f0848074390 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f083e7fc640 1 -- 172.21.15.167:0/1345272954 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0844004be0 con 0x7f0848074390 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f084fb6e640 1 -- 172.21.15.167:0/1345272954 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f084813f9a0 con 0x7f0848074390 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f083e7fc640 1 -- 172.21.15.167:0/1345272954 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0844011070 con 0x7f0848074390 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f083e7fc640 1 -- 172.21.15.167:0/1345272954 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f0844013a20 con 0x7f0848074390 2024-03-19T01:23:04.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.678+0000 7f082bfff640 1 -- 172.21.15.167:0/1345272954 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08240052a0 con 0x7f0848074390 2024-03-19T01:23:04.685 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.682+0000 7f083e7fc640 1 -- 172.21.15.167:0/1345272954 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0844013c20 con 0x7f0848074390 2024-03-19T01:23:04.749 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.746+0000 7f082bfff640 1 -- 172.21.15.167:0/1345272954 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1bdb920f-d25e-4092-9d3f-e2cabe5e21f3", "id": 7} v 0) -- 0x7f0824005860 con 0x7f0848074390 2024-03-19T01:23:04.755 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.750+0000 7f083e7fc640 1 -- 172.21.15.167:0/1345272954 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1bdb920f-d25e-4092-9d3f-e2cabe5e21f3", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f0844013e00 con 0x7f0848074390 2024-03-19T01:23:04.756 INFO:teuthology.orchestra.run.smithi167.stdout:7 2024-03-19T01:23:04.757 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.754+0000 7f082bfff640 1 -- 172.21.15.167:0/1345272954 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0848074390 msgr2=0x7f084813ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.757 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.754+0000 7f082bfff640 1 --2- 172.21.15.167:0/1345272954 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0848074390 0x7f084813ed50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f084400c430 tx=0x7f084400c8f0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.757 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.754+0000 7f082bfff640 1 -- 172.21.15.167:0/1345272954 shutdown_connections 2024-03-19T01:23:04.757 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.754+0000 7f082bfff640 1 --2- 172.21.15.167:0/1345272954 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0848140ac0 0x7f0848145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.757 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.754+0000 7f082bfff640 1 --2- 172.21.15.167:0/1345272954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08480ae810 0x7f0848145650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.757 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.754+0000 7f082bfff640 1 --2- 172.21.15.167:0/1345272954 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0848074390 0x7f084813ed50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.758 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.754+0000 7f082bfff640 1 -- 172.21.15.167:0/1345272954 >> 172.21.15.167:0/1345272954 conn(0x7f08480b5c90 msgr2=0x7f08480b87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:04.758 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.754+0000 7f082bfff640 1 -- 172.21.15.167:0/1345272954 shutdown_connections 2024-03-19T01:23:04.758 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-19T01:23:04.754+0000 7f082bfff640 1 -- 172.21.15.167:0/1345272954 wait complete. 2024-03-19T01:23:04.772 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-19T01:23:04.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-19T01:23:04.774 INFO:tasks.ceph.osd.0:Started 2024-03-19T01:23:04.774 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-19T01:23:04.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-19T01:23:04.777 INFO:tasks.ceph.osd.1:Started 2024-03-19T01:23:04.777 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-19T01:23:04.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-19T01:23:04.780 INFO:tasks.ceph.osd.2:Started 2024-03-19T01:23:04.780 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-19T01:23:04.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-19T01:23:04.782 INFO:tasks.ceph.osd.3:Started 2024-03-19T01:23:04.782 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-19T01:23:04.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-19T01:23:04.784 INFO:tasks.ceph.osd.4:Started 2024-03-19T01:23:04.784 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-19T01:23:04.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-19T01:23:04.786 INFO:tasks.ceph.osd.5:Started 2024-03-19T01:23:04.786 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-19T01:23:04.786 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-19T01:23:04.788 INFO:tasks.ceph.osd.6:Started 2024-03-19T01:23:04.788 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-19T01:23:04.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-19T01:23:04.790 INFO:tasks.ceph.osd.7:Started 2024-03-19T01:23:04.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:04.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb00d27c640 1 Processor -- start 2024-03-19T01:23:04.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb00d27c640 1 -- start start 2024-03-19T01:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb00d27c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0080b9f30 0x7fb0080b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb00d27c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b88c0 0x7fb0081451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb00d27c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0080b7960 con 0x7fb008072ba0 2024-03-19T01:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb00d27c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fb0080b8d50 con 0x7fb0080b9f30 2024-03-19T01:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb00d27c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb0080b7c40 con 0x7fb0080b88c0 2024-03-19T01:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb006ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0080b9f30 0x7fb0080b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb006ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0080b9f30 0x7fb0080b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40302/0 (socket says 172.21.15.7:40302) 2024-03-19T01:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb006ffd640 1 -- 172.21.15.7:0/2818481723 learned_addr learned my addr 172.21.15.7:0/2818481723 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb0067fc640 1 --2- 172.21.15.7:0/2818481723 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b88c0 0x7fb0081451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb006ffd640 1 -- 172.21.15.7:0/2818481723 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b88c0 msgr2=0x7fb0081451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:04.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb006ffd640 1 --2- 172.21.15.7:0/2818481723 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b88c0 0x7fb0081451e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb006ffd640 1 -- 172.21.15.7:0/2818481723 >> v1:172.21.15.167:6789/0 conn(0x7fb008072ba0 legacy=0x7fb008148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:04.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.909+0000 7fb006ffd640 1 -- 172.21.15.7:0/2818481723 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0080ba310 con 0x7fb0080b9f30 2024-03-19T01:23:04.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb006ffd640 1 --2- 172.21.15.7:0/2818481723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080b9f30 0x7fb0080b42a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb0080b8ed0 tx=0x7faff801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e7663160330ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb005ffb640 1 -- 172.21.15.7:0/2818481723 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff80b0070 con 0x7fb0080b9f30 2024-03-19T01:23:04.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb005ffb640 1 -- 172.21.15.7:0/2818481723 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff80ab600 con 0x7fb0080b9f30 2024-03-19T01:23:04.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb005ffb640 1 -- 172.21.15.7:0/2818481723 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff80ab9d0 con 0x7fb0080b9f30 2024-03-19T01:23:04.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 -- 172.21.15.7:0/2818481723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080b9f30 msgr2=0x7fb0080b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 --2- 172.21.15.7:0/2818481723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080b9f30 0x7fb0080b42a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb0080b8ed0 tx=0x7faff801c470 comp rx=0 tx=0).stop 2024-03-19T01:23:04.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 -- 172.21.15.7:0/2818481723 shutdown_connections 2024-03-19T01:23:04.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 --2- 172.21.15.7:0/2818481723 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b88c0 0x7fb0081451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 --2- 172.21.15.7:0/2818481723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080b9f30 0x7fb0080b42a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 -- 172.21.15.7:0/2818481723 >> 172.21.15.7:0/2818481723 conn(0x7fb00806b490 msgr2=0x7fb00806b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:04.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 -- 172.21.15.7:0/2818481723 shutdown_connections 2024-03-19T01:23:04.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 -- 172.21.15.7:0/2818481723 wait complete. 2024-03-19T01:23:04.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 Processor -- start 2024-03-19T01:23:04.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.913+0000 7fb00d27c640 1 -- start start 2024-03-19T01:23:04.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb00d27c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb008072ba0 0x7fb0081380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb00d27c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b9f30 0x7fb0081de8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb00d27c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb008146b50 0x7fb0081dee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:04.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb00d27c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb008152e70 con 0x7fb008146b50 2024-03-19T01:23:04.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb00d27c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb008152cf0 con 0x7fb008072ba0 2024-03-19T01:23:04.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb00d27c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb008152ff0 con 0x7fb0080b9f30 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0077fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb008146b50 0x7fb0081dee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0077fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb008146b50 0x7fb0081dee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40308/0 (socket says 172.21.15.7:40308) 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0077fe640 1 -- 172.21.15.7:0/2306714321 learned_addr learned my addr 172.21.15.7:0/2306714321 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb006ffd640 1 --2- 172.21.15.7:0/2306714321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb008072ba0 0x7fb0081380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0067fc640 1 --2- 172.21.15.7:0/2306714321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b9f30 0x7fb0081de8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0077fe640 1 -- 172.21.15.7:0/2306714321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b9f30 msgr2=0x7fb0081de8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0077fe640 1 --2- 172.21.15.7:0/2306714321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b9f30 0x7fb0081de8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0077fe640 1 -- 172.21.15.7:0/2306714321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb008072ba0 msgr2=0x7fb0081380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0077fe640 1 --2- 172.21.15.7:0/2306714321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb008072ba0 0x7fb0081380c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0077fe640 1 -- 172.21.15.7:0/2306714321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb008147ef0 con 0x7fb008146b50 2024-03-19T01:23:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb0077fe640 1 --2- 172.21.15.7:0/2306714321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb008146b50 0x7fb0081dee20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c7b0 tx=0x7fb00000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:04.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fafef7fe640 1 -- 172.21.15.7:0/2306714321 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000016020 con 0x7fb008146b50 2024-03-19T01:23:04.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fafef7fe640 1 -- 172.21.15.7:0/2306714321 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00000a9f0 con 0x7fb008146b50 2024-03-19T01:23:04.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb00d27c640 1 -- 172.21.15.7:0/2306714321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0081464e0 con 0x7fb008146b50 2024-03-19T01:23:04.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fafef7fe640 1 -- 172.21.15.7:0/2306714321 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00000ade0 con 0x7fb008146b50 2024-03-19T01:23:04.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb00d27c640 1 -- 172.21.15.7:0/2306714321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0081372c0 con 0x7fb008146b50 2024-03-19T01:23:04.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fafef7fe640 1 -- 172.21.15.7:0/2306714321 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb000019020 con 0x7fb008146b50 2024-03-19T01:23:04.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fafef7fe640 1 -- 172.21.15.7:0/2306714321 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fb00001a3b0 con 0x7fb008146b50 2024-03-19T01:23:04.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fb00d27c640 1 -- 172.21.15.7:0/2306714321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb008074130 con 0x7fb008146b50 2024-03-19T01:23:04.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:04.917+0000 7fafef7fe640 1 -- 172.21.15.7:0/2306714321 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb000020020 con 0x7fb008146b50 2024-03-19T01:23:04.929 INFO:tasks.ceph.mgr.y.smithi007.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-19T01:23:04.929 INFO:tasks.ceph.mgr.y.smithi007.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-19T01:23:04.929 INFO:tasks.ceph.mgr.y.smithi007.stderr: from numpy import show_config as show_numpy_config 2024-03-19T01:23:04.970 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:04.969+0000 7f198c2e3180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T01:23:05.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 -- 172.21.15.7:0/2306714321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb008075460 con 0x7fb008146b50 2024-03-19T01:23:05.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fafef7fe640 1 -- 172.21.15.7:0/2306714321 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fb000012990 con 0x7fb008146b50 2024-03-19T01:23:05.011 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:05.011 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:04.752249+0000","last_up_change":"0.000000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:05.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 -- 172.21.15.7:0/2306714321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb008146b50 msgr2=0x7fb0081dee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:05.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 --2- 172.21.15.7:0/2306714321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb008146b50 0x7fb0081dee20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c7b0 tx=0x7fb00000cc70 comp rx=0 tx=0).stop 2024-03-19T01:23:05.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 -- 172.21.15.7:0/2306714321 shutdown_connections 2024-03-19T01:23:05.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 --2- 172.21.15.7:0/2306714321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0080b9f30 0x7fb0081de8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:05.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 --2- 172.21.15.7:0/2306714321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb008146b50 0x7fb0081dee20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:05.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 --2- 172.21.15.7:0/2306714321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb008072ba0 0x7fb0081380c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:05.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 -- 172.21.15.7:0/2306714321 >> 172.21.15.7:0/2306714321 conn(0x7fb00806b490 msgr2=0x7fb00806b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:05.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 -- 172.21.15.7:0/2306714321 shutdown_connections 2024-03-19T01:23:05.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:05.009+0000 7fb00d27c640 1 -- 172.21.15.7:0/2306714321 wait complete. 2024-03-19T01:23:05.028 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-19T01:23:05.028 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-19T01:23:05.126 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:05.125+0000 7f198c2e3180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:23:05.278 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:05.277+0000 7f198c2e3180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T01:23:05.299 INFO:tasks.ceph.mgr.x.smithi167.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-19T01:23:05.299 INFO:tasks.ceph.mgr.x.smithi167.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-19T01:23:05.299 INFO:tasks.ceph.mgr.x.smithi167.stderr: from numpy import show_config as show_numpy_config 2024-03-19T01:23:05.344 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:05.341+0000 7f198c2e3180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T01:23:05.367 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:05.362+0000 7fc8b1fa5180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T01:23:05.387 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-19T01:23:05.385+0000 7fa914f6f7c0 -1 Falling back to public interface 2024-03-19T01:23:05.390 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-03-19T01:23:05.386+0000 7fb81701d7c0 -1 Falling back to public interface 2024-03-19T01:23:05.391 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-03-19T01:23:05.386+0000 7fcab743e7c0 -1 Falling back to public interface 2024-03-19T01:23:05.391 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-03-19T01:23:05.386+0000 7f99b6adc7c0 -1 Falling back to public interface 2024-03-19T01:23:05.391 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-03-19T01:23:05.386+0000 7fcab63fe640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6802/0: (98) Address already in use 2024-03-19T01:23:05.402 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-19T01:23:05.401+0000 7f5fcd3337c0 -1 Falling back to public interface 2024-03-19T01:23:05.414 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-19T01:23:05.413+0000 7fb9c41b67c0 -1 Falling back to public interface 2024-03-19T01:23:05.418 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-19T01:23:05.417+0000 7fe8f31f37c0 -1 Falling back to public interface 2024-03-19T01:23:05.421 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-03-19T01:23:05.418+0000 7fcde4ece7c0 -1 Falling back to public interface 2024-03-19T01:23:05.487 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:05.485+0000 7f198c2e3180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:23:05.540 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:05.534+0000 7fc8b1fa5180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:23:05.713 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:05.710+0000 7fc8b1fa5180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T01:23:05.783 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:05.778+0000 7fc8b1fa5180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T01:23:05.832 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:05.930 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:05.926+0000 7fc8b1fa5180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:23:05.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.945+0000 7f4bb45d2640 1 Processor -- start 2024-03-19T01:23:05.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.945+0000 7f4bb45d2640 1 -- start start 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4bac14a9a0 0x7f4bac149c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac09f5a0 con 0x7f4bac0671e0 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac0675c0 con 0x7f4bac14ae00 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f4bac0aef50 con 0x7f4bac14a9a0 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb1b6d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4bac14a9a0 0x7f4bac149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb236e640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4bac0671e0 0x7f4bac0b4d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49690/0 (socket says 172.21.15.7:49690) 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb236e640 1 -- 172.21.15.7:0/3627474526 learned_addr learned my addr 172.21.15.7:0/3627474526 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb136c640 1 -- 172.21.15.7:0/3627474526 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4094843208 0 0) 0x7f4bac09f5a0 con 0x7f4bac0671e0 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb136c640 1 -- 172.21.15.7:0/3627474526 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b98003530 con 0x7f4bac0671e0 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb1b6d640 1 -- 172.21.15.7:0/3627474526 >> v1:172.21.15.167:6789/0 conn(0x7f4bac14ae00 legacy=0x7f4bac14a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb1b6d640 1 -- 172.21.15.7:0/3627474526 >> v1:172.21.15.7:6789/0 conn(0x7f4bac0671e0 legacy=0x7f4bac0b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:05.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb1b6d640 1 -- 172.21.15.7:0/3627474526 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac09f840 con 0x7f4bac14a9a0 2024-03-19T01:23:05.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb1b6d640 1 --2- 172.21.15.7:0/3627474526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 0x7f4bac149c70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c0097c0 tx=0x7f4b9c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248e974f88a61447 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:05.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb136c640 1 -- 172.21.15.7:0/3627474526 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c0b0070 con 0x7f4bac14a9a0 2024-03-19T01:23:05.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb136c640 1 -- 172.21.15.7:0/3627474526 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b9c0ab600 con 0x7f4bac14a9a0 2024-03-19T01:23:05.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb136c640 1 -- 172.21.15.7:0/3627474526 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c0ab9d0 con 0x7f4bac14a9a0 2024-03-19T01:23:05.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 -- 172.21.15.7:0/3627474526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 msgr2=0x7f4bac149c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:05.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 --2- 172.21.15.7:0/3627474526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 0x7f4bac149c70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c0097c0 tx=0x7f4b9c01c3d0 comp rx=0 tx=0).stop 2024-03-19T01:23:05.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 -- 172.21.15.7:0/3627474526 shutdown_connections 2024-03-19T01:23:05.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 --2- 172.21.15.7:0/3627474526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 0x7f4bac149c70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:05.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 -- 172.21.15.7:0/3627474526 >> 172.21.15.7:0/3627474526 conn(0x7f4bac0b9a50 msgr2=0x7f4bac0b9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:05.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 -- 172.21.15.7:0/3627474526 shutdown_connections 2024-03-19T01:23:05.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.949+0000 7f4bb45d2640 1 -- 172.21.15.7:0/3627474526 wait complete. 2024-03-19T01:23:05.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 Processor -- start 2024-03-19T01:23:05.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 -- start start 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bac0671e0 0x7f4bac1dfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 0x7f4bac0b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bac14ae00 0x7f4bac1e3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac155680 con 0x7f4bac14a9a0 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac155500 con 0x7f4bac14ae00 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4bac155800 con 0x7f4bac0671e0 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb1b6d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 0x7f4bac0b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb1b6d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 0x7f4bac0b1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40322/0 (socket says 172.21.15.7:40322) 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb1b6d640 1 -- 172.21.15.7:0/1943991468 learned_addr learned my addr 172.21.15.7:0/1943991468 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb2b6f640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bac14ae00 0x7f4bac1e3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb236e640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bac0671e0 0x7f4bac1dfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb1b6d640 1 -- 172.21.15.7:0/1943991468 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bac0671e0 msgr2=0x7f4bac1dfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb1b6d640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bac0671e0 0x7f4bac1dfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:05.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb1b6d640 1 -- 172.21.15.7:0/1943991468 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bac14ae00 msgr2=0x7f4bac1e3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb1b6d640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bac14ae00 0x7f4bac1e3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb1b6d640 1 -- 172.21.15.7:0/1943991468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac0b33f0 con 0x7f4bac14a9a0 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb236e640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bac0671e0 0x7f4bac1dfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb1b6d640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 0x7f4bac0b1e10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c018fe0 tx=0x7f4b9c01ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4ba2ffd640 1 -- 172.21.15.7:0/1943991468 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c0b0050 con 0x7f4bac14a9a0 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4ba2ffd640 1 -- 172.21.15.7:0/1943991468 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b9c0bc070 con 0x7f4bac14a9a0 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 -- 172.21.15.7:0/1943991468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bac1405a0 con 0x7f4bac14a9a0 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4ba2ffd640 1 -- 172.21.15.7:0/1943991468 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c0b5830 con 0x7f4bac14a9a0 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 -- 172.21.15.7:0/1943991468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bac1e00e0 con 0x7f4bac14a9a0 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4ba2ffd640 1 -- 172.21.15.7:0/1943991468 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4b9c0b59d0 con 0x7f4bac14a9a0 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4ba2ffd640 1 -- 172.21.15.7:0/1943991468 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f4b9c0b7b20 con 0x7f4bac14a9a0 2024-03-19T01:23:05.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.953+0000 7f4bb45d2640 1 -- 172.21.15.7:0/1943991468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bac04f590 con 0x7f4bac14a9a0 2024-03-19T01:23:05.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:05.957+0000 7f4ba2ffd640 1 -- 172.21.15.7:0/1943991468 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4b9c016050 con 0x7f4bac14a9a0 2024-03-19T01:23:06.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.013+0000 7f4bb45d2640 1 -- 172.21.15.7:0/1943991468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4bac071e80 con 0x7f4bac14a9a0 2024-03-19T01:23:06.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.013+0000 7f4ba2ffd640 1 -- 172.21.15.7:0/1943991468 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f4b9c0b5cc0 con 0x7f4bac14a9a0 2024-03-19T01:23:06.018 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-19T01:23:06.018 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:04.752249+0000","last_up_change":"0.000000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:06.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.017+0000 7f4bb45d2640 1 -- 172.21.15.7:0/1943991468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 msgr2=0x7f4bac0b1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:06.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.017+0000 7f4bb45d2640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 0x7f4bac0b1e10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c018fe0 tx=0x7f4b9c01ac10 comp rx=0 tx=0).stop 2024-03-19T01:23:06.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.017+0000 7f4bb45d2640 1 -- 172.21.15.7:0/1943991468 shutdown_connections 2024-03-19T01:23:06.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.017+0000 7f4bb45d2640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bac14ae00 0x7f4bac1e3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.017+0000 7f4bb45d2640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bac14a9a0 0x7f4bac0b1e10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.017+0000 7f4bb45d2640 1 --2- 172.21.15.7:0/1943991468 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bac0671e0 0x7f4bac1dfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.017+0000 7f4bb45d2640 1 -- 172.21.15.7:0/1943991468 >> 172.21.15.7:0/1943991468 conn(0x7f4bac0b9a50 msgr2=0x7f4bac0afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:06.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.017+0000 7f4bb45d2640 1 -- 172.21.15.7:0/1943991468 shutdown_connections 2024-03-19T01:23:06.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:06.017+0000 7f4bb45d2640 1 -- 172.21.15.7:0/1943991468 wait complete. 2024-03-19T01:23:06.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-19T01:23:06.136 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:06.133+0000 7f198c2e3180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T01:23:06.214 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:06.213+0000 7f198c2e3180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T01:23:06.404 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:06.401+0000 7f198c2e3180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T01:23:06.481 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:06.477+0000 7f198c2e3180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T01:23:06.587 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:06.582+0000 7fc8b1fa5180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T01:23:06.662 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:06.658+0000 7fc8b1fa5180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T01:23:06.786 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:06.785+0000 7f198c2e3180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T01:23:06.842 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:06.838+0000 7fc8b1fa5180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T01:23:06.851 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:06.849+0000 7f198c2e3180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T01:23:06.920 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:06.914+0000 7fc8b1fa5180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T01:23:07.209 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:07.206+0000 7fc8b1fa5180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T01:23:07.270 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-19T01:23:07.266+0000 7fc8b1fa5180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T01:23:07.904 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:07.893+0000 7f1933629640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-19T01:23:07.905 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-19T01:23:07.893+0000 7f1933629640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-19T01:23:09.003 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-03-19T01:23:08.997+0000 7fcab743e7c0 -1 osd.6 0 log_to_monitors true 2024-03-19T01:23:09.020 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-03-19T01:23:09.017+0000 7fb81701d7c0 -1 osd.4 0 log_to_monitors true 2024-03-19T01:23:09.139 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-19T01:23:09.137+0000 7fa914f6f7c0 -1 osd.0 0 log_to_monitors true 2024-03-19T01:23:09.187 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-03-19T01:23:09.181+0000 7f99b6adc7c0 -1 osd.7 0 log_to_monitors true 2024-03-19T01:23:09.249 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-19T01:23:09.245+0000 7f5fcd3337c0 -1 osd.1 0 log_to_monitors true 2024-03-19T01:23:09.264 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-19T01:23:09.261+0000 7fb9c41b67c0 -1 osd.2 0 log_to_monitors true 2024-03-19T01:23:09.298 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-03-19T01:23:09.293+0000 7fcde4ece7c0 -1 osd.5 0 log_to_monitors true 2024-03-19T01:23:09.360 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-19T01:23:09.357+0000 7fe8f31f37c0 -1 osd.3 0 log_to_monitors true 2024-03-19T01:23:12.390 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-19T01:23:12.384+0000 7fe8ef19c640 -1 osd.3 0 waiting for initial osdmap 2024-03-19T01:23:12.419 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-19T01:23:12.416+0000 7fe8e6fd1640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:23:12.498 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-19T01:23:12.492+0000 7fb9c015f640 -1 osd.2 0 waiting for initial osdmap 2024-03-19T01:23:12.520 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-19T01:23:12.516+0000 7f5fc92dc640 -1 osd.1 0 waiting for initial osdmap 2024-03-19T01:23:12.527 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-19T01:23:12.524+0000 7fb9b7f94640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:23:12.547 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-19T01:23:12.544+0000 7f5fc1111640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:23:12.559 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-19T01:23:12.556+0000 7fa910f18640 -1 osd.0 0 waiting for initial osdmap 2024-03-19T01:23:12.594 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-19T01:23:12.592+0000 7fa908d4d640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:23:12.838 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:12.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc9c653640 1 Processor -- start 2024-03-19T01:23:12.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc9c653640 1 -- start start 2024-03-19T01:23:12.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc9c653640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc940b1470 0x7efc94148a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:12.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc9c653640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc940b72b0 con 0x7efc940b18f0 2024-03-19T01:23:12.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc9c653640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc940b5f90 con 0x7efc9406e5b0 2024-03-19T01:23:12.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc9c653640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7efc940b7520 con 0x7efc940b1470 2024-03-19T01:23:12.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc99bee640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc940b1470 0x7efc94148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:12.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc9abf0640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efc940b18f0 0x7efc9414fc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52862/0 (socket says 172.21.15.7:52862) 2024-03-19T01:23:12.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc9abf0640 1 -- 172.21.15.7:0/1727598840 learned_addr learned my addr 172.21.15.7:0/1727598840 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:12.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc99bee640 1 -- 172.21.15.7:0/1727598840 >> v1:172.21.15.167:6789/0 conn(0x7efc9406e5b0 legacy=0x7efc9406e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:12.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc99bee640 1 -- 172.21.15.7:0/1727598840 >> v1:172.21.15.7:6789/0 conn(0x7efc940b18f0 legacy=0x7efc9414fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:12.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc99bee640 1 -- 172.21.15.7:0/1727598840 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc940b6270 con 0x7efc940b1470 2024-03-19T01:23:12.966 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc99bee640 1 --2- 172.21.15.7:0/1727598840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc940b1470 0x7efc94148a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efc840097c0 tx=0x7efc8401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a567725dc1eff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:12.966 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc993ed640 1 -- 172.21.15.7:0/1727598840 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc840b0070 con 0x7efc940b1470 2024-03-19T01:23:12.966 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc993ed640 1 -- 172.21.15.7:0/1727598840 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc840ab600 con 0x7efc940b1470 2024-03-19T01:23:12.966 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.960+0000 7efc993ed640 1 -- 172.21.15.7:0/1727598840 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc840ab9d0 con 0x7efc940b1470 2024-03-19T01:23:12.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- 172.21.15.7:0/1727598840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc940b1470 msgr2=0x7efc94148a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:12.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 --2- 172.21.15.7:0/1727598840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc940b1470 0x7efc94148a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efc840097c0 tx=0x7efc8401c410 comp rx=0 tx=0).stop 2024-03-19T01:23:12.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- 172.21.15.7:0/1727598840 shutdown_connections 2024-03-19T01:23:12.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 --2- 172.21.15.7:0/1727598840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc940b1470 0x7efc94148a60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:12.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- 172.21.15.7:0/1727598840 >> 172.21.15.7:0/1727598840 conn(0x7efc940bada0 msgr2=0x7efc940bb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:12.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- 172.21.15.7:0/1727598840 shutdown_connections 2024-03-19T01:23:12.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- 172.21.15.7:0/1727598840 wait complete. 2024-03-19T01:23:12.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 Processor -- start 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- start start 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc9406e5b0 0x7efc9409f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc940b1470 0x7efc941e0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940b18f0 0x7efc94131ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efc94153ef0 con 0x7efc9406e5b0 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efc94153d70 con 0x7efc940b18f0 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efc94154070 con 0x7efc940b1470 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9a3ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc9406e5b0 0x7efc9409f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9a3ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc9406e5b0 0x7efc9409f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42096/0 (socket says 172.21.15.7:42096) 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9a3ef640 1 -- 172.21.15.7:0/3935754280 learned_addr learned my addr 172.21.15.7:0/3935754280 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:12.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9abf0640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940b18f0 0x7efc94131ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc99bee640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc940b1470 0x7efc941e0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9a3ef640 1 -- 172.21.15.7:0/3935754280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc940b1470 msgr2=0x7efc941e0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9a3ef640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc940b1470 0x7efc941e0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9a3ef640 1 -- 172.21.15.7:0/3935754280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940b18f0 msgr2=0x7efc94131ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9a3ef640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940b18f0 0x7efc94131ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9a3ef640 1 -- 172.21.15.7:0/3935754280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc94137f30 con 0x7efc9406e5b0 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9a3ef640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc9406e5b0 0x7efc9409f0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efc9000c3a0 tx=0x7efc9000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc8affd640 1 -- 172.21.15.7:0/3935754280 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc900052d0 con 0x7efc9406e5b0 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc8affd640 1 -- 172.21.15.7:0/3935754280 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc90005470 con 0x7efc9406e5b0 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc94136630 con 0x7efc9406e5b0 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc8affd640 1 -- 172.21.15.7:0/3935754280 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90007260 con 0x7efc9406e5b0 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc94136bf0 con 0x7efc9406e5b0 2024-03-19T01:23:12.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.964+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc9404f590 con 0x7efc9406e5b0 2024-03-19T01:23:12.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.968+0000 7efc8affd640 1 -- 172.21.15.7:0/3935754280 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) ==== 87027+0+0 (secure 0 0 0) 0x7efc90007400 con 0x7efc9406e5b0 2024-03-19T01:23:12.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.968+0000 7efc8affd640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efc780675d0 0x7efc78069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:12.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.968+0000 7efc8affd640 1 -- 172.21.15.7:0/3935754280 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7efc90083eb0 con 0x7efc9406e5b0 2024-03-19T01:23:12.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.968+0000 7efc99bee640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efc780675d0 0x7efc78069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:12.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.968+0000 7efc99bee640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efc780675d0 0x7efc78069a90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efc840ad000 tx=0x7efc84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:12.975 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:12.972+0000 7efc8affd640 1 -- 172.21.15.7:0/3935754280 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7efc90055990 con 0x7efc9406e5b0 2024-03-19T01:23:12.990 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-03-19T01:23:12.985+0000 7fb812fc6640 -1 osd.4 0 waiting for initial osdmap 2024-03-19T01:23:13.021 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-03-19T01:23:13.017+0000 7fb80adfb640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:23:13.021 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-03-19T01:23:13.017+0000 7fcde1689640 -1 osd.5 0 waiting for initial osdmap 2024-03-19T01:23:13.034 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-03-19T01:23:13.029+0000 7fcab33e7640 -1 osd.6 0 waiting for initial osdmap 2024-03-19T01:23:13.041 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-03-19T01:23:13.037+0000 7fcdd8cac640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:23:13.065 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-03-19T01:23:13.061+0000 7fcaab21c640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:23:13.083 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-03-19T01:23:13.077+0000 7f99b2a85640 -1 osd.7 0 waiting for initial osdmap 2024-03-19T01:23:13.105 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-03-19T01:23:13.101+0000 7f99aa8ba640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:23:13.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.128+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc941480b0 con 0x7efc9406e5b0 2024-03-19T01:23:13.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.128+0000 7efc8affd640 1 -- 172.21.15.7:0/3935754280 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7efc90055290 con 0x7efc9406e5b0 2024-03-19T01:23:13.134 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-19T01:23:13.135 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":12,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:10.899219+0000","last_up_change":"0.000000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efc780675d0 msgr2=0x7efc78069a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efc780675d0 0x7efc78069a90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efc840ad000 tx=0x7efc84008040 comp rx=0 tx=0).stop 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc9406e5b0 msgr2=0x7efc9409f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc9406e5b0 0x7efc9409f0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efc9000c3a0 tx=0x7efc9000c860 comp rx=0 tx=0).stop 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 shutdown_connections 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efc780675d0 0x7efc78069a90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc940b1470 0x7efc941e0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940b18f0 0x7efc94131ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 --2- 172.21.15.7:0/3935754280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc9406e5b0 0x7efc9409f0d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 >> 172.21.15.7:0/3935754280 conn(0x7efc940bada0 msgr2=0x7efc94071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 shutdown_connections 2024-03-19T01:23:13.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:13.132+0000 7efc9c653640 1 -- 172.21.15.7:0/3935754280 wait complete. 2024-03-19T01:23:13.150 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-19T01:23:19.957 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:20.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f0592826640 1 Processor -- start 2024-03-19T01:23:20.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f0592826640 1 -- start start 2024-03-19T01:23:20.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f0592826640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f058c072690 0x7f058c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f0592826640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f058c14fd60 0x7f058c152150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f0592826640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058c0a03c0 con 0x7f058c0b6a30 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f0592826640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f058c0a0170 con 0x7f058c072690 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f0592826640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f058c072a70 con 0x7f058c14fd60 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f058bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f058c072690 0x7f058c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f058bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f058c072690 0x7f058c06ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42134/0 (socket says 172.21.15.7:42134) 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f058bfff640 1 -- 172.21.15.7:0/256013482 learned_addr learned my addr 172.21.15.7:0/256013482 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f0590dc3640 1 --2- 172.21.15.7:0/256013482 >> v2:172.21.15.167:3300/0 conn(0x7f058c14fd60 0x7f058c152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f058bfff640 1 -- 172.21.15.7:0/256013482 >> v2:172.21.15.167:3300/0 conn(0x7f058c14fd60 msgr2=0x7f058c152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f058bfff640 1 --2- 172.21.15.7:0/256013482 >> v2:172.21.15.167:3300/0 conn(0x7f058c14fd60 0x7f058c152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f058bfff640 1 -- 172.21.15.7:0/256013482 >> v1:172.21.15.167:6789/0 conn(0x7f058c0b6a30 legacy=0x7f058c141500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f058bfff640 1 -- 172.21.15.7:0/256013482 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c141c10 con 0x7f058c072690 2024-03-19T01:23:20.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.080+0000 7f0590dc3640 1 --2- 172.21.15.7:0/256013482 >> v2:172.21.15.167:3300/0 conn(0x7f058c14fd60 0x7f058c152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:20.086 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 --2- 172.21.15.7:0/256013482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 0x7f058c06ecd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f058c0a02f0 tx=0x7f057c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2074258ad0c95e1f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.087 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058b7fe640 1 -- 172.21.15.7:0/256013482 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0b0070 con 0x7f058c072690 2024-03-19T01:23:20.087 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058b7fe640 1 -- 172.21.15.7:0/256013482 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f057c0ab600 con 0x7f058c072690 2024-03-19T01:23:20.087 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058b7fe640 1 -- 172.21.15.7:0/256013482 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0ab9d0 con 0x7f058c072690 2024-03-19T01:23:20.087 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 -- 172.21.15.7:0/256013482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 msgr2=0x7f058c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.087 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 --2- 172.21.15.7:0/256013482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 0x7f058c06ecd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f058c0a02f0 tx=0x7f057c01c490 comp rx=0 tx=0).stop 2024-03-19T01:23:20.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 -- 172.21.15.7:0/256013482 shutdown_connections 2024-03-19T01:23:20.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 --2- 172.21.15.7:0/256013482 >> v2:172.21.15.167:3300/0 conn(0x7f058c14fd60 0x7f058c152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 --2- 172.21.15.7:0/256013482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 0x7f058c06ecd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 -- 172.21.15.7:0/256013482 >> 172.21.15.7:0/256013482 conn(0x7f058c0b77a0 msgr2=0x7f058c0b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 -- 172.21.15.7:0/256013482 shutdown_connections 2024-03-19T01:23:20.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 -- 172.21.15.7:0/256013482 wait complete. 2024-03-19T01:23:20.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 Processor -- start 2024-03-19T01:23:20.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 -- start start 2024-03-19T01:23:20.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 0x7f058c1513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f058c0b6a30 0x7f058c142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f058c14fd60 0x7f058c1477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f058c0700f0 con 0x7f058c072690 2024-03-19T01:23:20.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f058c06ff70 con 0x7f058c0b6a30 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f0592826640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f058c0a05e0 con 0x7f058c14fd60 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 0x7f058c1513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 0x7f058c1513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42138/0 (socket says 172.21.15.7:42138) 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 -- 172.21.15.7:0/3715174955 learned_addr learned my addr 172.21.15.7:0/3715174955 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 -- 172.21.15.7:0/3715174955 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f058c14fd60 msgr2=0x7f058c1477c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f058c14fd60 0x7f058c1477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 -- 172.21.15.7:0/3715174955 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f058c0b6a30 msgr2=0x7f058c142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f058c0b6a30 0x7f058c142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 -- 172.21.15.7:0/3715174955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c142b80 con 0x7f058c072690 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f058bfff640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 0x7f058c1513f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f057c0ae040 tx=0x7f057c019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f05897fa640 1 -- 172.21.15.7:0/3715174955 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0b0040 con 0x7f058c072690 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.084+0000 7f05897fa640 1 -- 172.21.15.7:0/3715174955 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f057c0b9070 con 0x7f058c072690 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.088+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058c142e00 con 0x7f058c072690 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.088+0000 7f05897fa640 1 -- 172.21.15.7:0/3715174955 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0b4660 con 0x7f058c072690 2024-03-19T01:23:20.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.088+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058c151e10 con 0x7f058c072690 2024-03-19T01:23:20.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.088+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f058c146dd0 con 0x7f058c072690 2024-03-19T01:23:20.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.088+0000 7f05897fa640 1 -- 172.21.15.7:0/3715174955 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f057c0b4800 con 0x7f058c072690 2024-03-19T01:23:20.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.088+0000 7f05897fa640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f056c067660 0x7f056c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.088+0000 7f05897fa640 1 -- 172.21.15.7:0/3715174955 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f057c11f190 con 0x7f058c072690 2024-03-19T01:23:20.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.088+0000 7f0583fff640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f056c067660 0x7f056c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.088+0000 7f0583fff640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f056c067660 0x7f056c069b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0574007fd0 tx=0x7f0574007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.092+0000 7f05897fa640 1 -- 172.21.15.7:0/3715174955 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f058c142b80 con 0x7f058c072690 2024-03-19T01:23:20.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.256+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f058c04f590 con 0x7f058c072690 2024-03-19T01:23:20.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.256+0000 7f05897fa640 1 -- 172.21.15.7:0/3715174955 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7f057c016110 con 0x7f058c072690 2024-03-19T01:23:20.262 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-19T01:23:20.262 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":17,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:17.421890+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:20.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f056c067660 msgr2=0x7f056c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f056c067660 0x7f056c069b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0574007fd0 tx=0x7f0574007980 comp rx=0 tx=0).stop 2024-03-19T01:23:20.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 msgr2=0x7f058c1513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 0x7f058c1513f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f057c0ae040 tx=0x7f057c019230 comp rx=0 tx=0).stop 2024-03-19T01:23:20.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 shutdown_connections 2024-03-19T01:23:20.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f056c067660 0x7f056c069b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f058c14fd60 0x7f058c1477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f058c0b6a30 0x7f058c142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 --2- 172.21.15.7:0/3715174955 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f058c072690 0x7f058c1513f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 >> 172.21.15.7:0/3715174955 conn(0x7f058c0b77a0 msgr2=0x7f058c13bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 shutdown_connections 2024-03-19T01:23:20.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-19T01:23:20.260+0000 7f0592826640 1 -- 172.21.15.7:0/3715174955 wait complete. 2024-03-19T01:23:20.280 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-19T01:23:20.280 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-19T01:23:20.280 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-19T01:23:20.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-19T01:23:20.283 INFO:tasks.ceph.mds.a:Started 2024-03-19T01:23:20.283 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-19T01:23:20.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-19T01:23:20.285 INFO:tasks.ceph.mds.b:Started 2024-03-19T01:23:20.286 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-19T01:23:20.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:20.355 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2024-03-19T01:23:20.386 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-03-19T01:23:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f02958ea640 1 Processor -- start 2024-03-19T01:23:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f02958ea640 1 -- start start 2024-03-19T01:23:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f02958ea640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f02900b81c0 0x7f02901486e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f02958ea640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02900b85a0 con 0x7f029006f640 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f02958ea640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02900b20c0 con 0x7f029006f1c0 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f02958ea640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f02900b2710 con 0x7f02900b81c0 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f028f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f029006f640 0x7f029014fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52886/0 (socket says 172.21.15.7:52886) 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f028f7fe640 1 -- 172.21.15.7:0/1049959868 learned_addr learned my addr 172.21.15.7:0/1049959868 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f028effd640 1 --2- 172.21.15.7:0/1049959868 >> v2:172.21.15.167:3300/0 conn(0x7f02900b81c0 0x7f02901486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3336881789 0 0) 0x7f02900b85a0 con 0x7f029006f640 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.396+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0278003530 con 0x7f029006f640 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 938185056 0 0) 0x7f0278003530 con 0x7f029006f640 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 >> v2:172.21.15.167:3300/0 conn(0x7f02900b81c0 msgr2=0x7f02901486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 --2- 172.21.15.7:0/1049959868 >> v2:172.21.15.167:3300/0 conn(0x7f02900b81c0 0x7f02901486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 >> v1:172.21.15.167:6789/0 conn(0x7f029006f1c0 legacy=0x7f029013f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02900b23a0 con 0x7f029006f640 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f0288001a00 con 0x7f029006f640 2024-03-19T01:23:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 >> v1:172.21.15.7:6789/0 conn(0x7f029006f640 legacy=0x7f029014fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02780039c0 0x7f0278003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0278004810 0x7f0278004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02780051d0 0x7f027800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f02900b2710 con 0x7f0278004810 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f027800c170 con 0x7f02780051d0 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f027800c370 con 0x7f02780039c0 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028e7fc640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0278004810 0x7f0278004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028effd640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02780039c0 0x7f0278003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028f7fe640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02780051d0 0x7f027800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028e7fc640 1 -- 172.21.15.7:0/1049959868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02780039c0 msgr2=0x7f0278003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028e7fc640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02780039c0 0x7f0278003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028e7fc640 1 -- 172.21.15.7:0/1049959868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02780051d0 msgr2=0x7f027800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028e7fc640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02780051d0 0x7f027800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028e7fc640 1 -- 172.21.15.7:0/1049959868 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0278003530 con 0x7f0278004810 2024-03-19T01:23:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028f7fe640 1 -- 172.21.15.7:0/1049959868 reap_dead start 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028e7fc640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0278004810 0x7f0278004c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0284001780 tx=0x7f028401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa4f3927c53b303 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840013a0 con 0x7f0278004810 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02840052f0 con 0x7f0278004810 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f028dffb640 1 -- 172.21.15.7:0/1049959868 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284003840 con 0x7f0278004810 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f02958ea640 1 -- 172.21.15.7:0/1049959868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0278004810 msgr2=0x7f0278004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f02958ea640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0278004810 0x7f0278004c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0284001780 tx=0x7f028401c400 comp rx=0 tx=0).stop 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f02958ea640 1 -- 172.21.15.7:0/1049959868 shutdown_connections 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f02958ea640 1 --2- 172.21.15.7:0/1049959868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0278004810 0x7f0278004c90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f02958ea640 1 -- 172.21.15.7:0/1049959868 >> 172.21.15.7:0/1049959868 conn(0x7f02900b5330 msgr2=0x7f02900b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f02958ea640 1 -- 172.21.15.7:0/1049959868 shutdown_connections 2024-03-19T01:23:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.400+0000 7f02958ea640 1 -- 172.21.15.7:0/1049959868 wait complete. 2024-03-19T01:23:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 Processor -- start 2024-03-19T01:23:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 -- start start 2024-03-19T01:23:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0278004810 0x7f0290132170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02900b4ee0 0x7f02901326b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02901de3c0 0x7f02901de840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f02901452f0 con 0x7f02900b4ee0 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0290145170 con 0x7f0278004810 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0290145470 con 0x7f02901de3c0 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02900b4ee0 0x7f02901326b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02900b4ee0 0x7f02901326b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42176/0 (socket says 172.21.15.7:42176) 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028e7fc640 1 -- 172.21.15.7:0/1661121662 learned_addr learned my addr 172.21.15.7:0/1661121662 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028effd640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0278004810 0x7f0290132170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028f7fe640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02901de3c0 0x7f02901de840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028e7fc640 1 -- 172.21.15.7:0/1661121662 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02901de3c0 msgr2=0x7f02901de840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028e7fc640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02901de3c0 0x7f02901de840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028e7fc640 1 -- 172.21.15.7:0/1661121662 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0278004810 msgr2=0x7f0290132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028e7fc640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0278004810 0x7f0290132170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028e7fc640 1 -- 172.21.15.7:0/1661121662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02900b36b0 con 0x7f02900b4ee0 2024-03-19T01:23:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f028e7fc640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02900b4ee0 0x7f02901326b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0284005040 tx=0x7f0278005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f0277fff640 1 -- 172.21.15.7:0/1661121662 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840b8020 con 0x7f02900b4ee0 2024-03-19T01:23:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f0277fff640 1 -- 172.21.15.7:0/1661121662 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02840054a0 con 0x7f02900b4ee0 2024-03-19T01:23:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0290132bf0 con 0x7f02900b4ee0 2024-03-19T01:23:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f0277fff640 1 -- 172.21.15.7:0/1661121662 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840013a0 con 0x7f02900b4ee0 2024-03-19T01:23:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02900b32d0 con 0x7f02900b4ee0 2024-03-19T01:23:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.404+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0290075460 con 0x7f02900b4ee0 2024-03-19T01:23:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.408+0000 7f0277fff640 1 -- 172.21.15.7:0/1661121662 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f02840bb020 con 0x7f02900b4ee0 2024-03-19T01:23:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.408+0000 7f0277fff640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f027c067660 0x7f027c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.408+0000 7f028effd640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f027c067660 0x7f027c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.408+0000 7f0277fff640 1 -- 172.21.15.7:0/1661121662 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f028411f730 con 0x7f02900b4ee0 2024-03-19T01:23:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.408+0000 7f028effd640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f027c067660 0x7f027c069b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0290138740 tx=0x7f0280009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.412+0000 7f0277fff640 1 -- 172.21.15.7:0/1661121662 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f02840c2030 con 0x7f02900b4ee0 2024-03-19T01:23:20.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.572+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0290139400 con 0x7f02900b4ee0 2024-03-19T01:23:20.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.572+0000 7f0277fff640 1 -- 172.21.15.7:0/1661121662 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7f0284016110 con 0x7f02900b4ee0 2024-03-19T01:23:20.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:20.577 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:17.421890+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f027c067660 msgr2=0x7f027c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f027c067660 0x7f027c069b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0290138740 tx=0x7f0280009190 comp rx=0 tx=0).stop 2024-03-19T01:23:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02900b4ee0 msgr2=0x7f02901326b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02900b4ee0 0x7f02901326b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0284005040 tx=0x7f0278005840 comp rx=0 tx=0).stop 2024-03-19T01:23:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 shutdown_connections 2024-03-19T01:23:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f027c067660 0x7f027c069b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02901de3c0 0x7f02901de840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02900b4ee0 0x7f02901326b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 --2- 172.21.15.7:0/1661121662 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0278004810 0x7f0290132170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 >> 172.21.15.7:0/1661121662 conn(0x7f02900b5330 msgr2=0x7f0290145800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 shutdown_connections 2024-03-19T01:23:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.576+0000 7f02958ea640 1 -- 172.21.15.7:0/1661121662 wait complete. 2024-03-19T01:23:20.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:20.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:20.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.712+0000 7f7d33052640 1 Processor -- start 2024-03-19T01:23:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.712+0000 7f7d33052640 1 -- start start 2024-03-19T01:23:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d33052640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7d2c149340 0x7f7d2c150a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d33052640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d2c0b5d80 con 0x7f7d2c06ff90 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d33052640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d2c0704c0 con 0x7f7d2c0b59a0 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d33052640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f7d2c072a80 con 0x7f7d2c149340 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d315ef640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7d2c149340 0x7f7d2c150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d2bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7d2c06ff90 0x7f7d2c071f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52896/0 (socket says 172.21.15.7:52896) 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d2bfff640 1 -- 172.21.15.7:0/4220174572 learned_addr learned my addr 172.21.15.7:0/4220174572 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d315ef640 1 -- 172.21.15.7:0/4220174572 >> v1:172.21.15.167:6789/0 conn(0x7f7d2c0b59a0 legacy=0x7f7d2c1450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d315ef640 1 -- 172.21.15.7:0/4220174572 >> v1:172.21.15.7:6789/0 conn(0x7f7d2c06ff90 legacy=0x7f7d2c071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d2b7fe640 1 -- 172.21.15.7:0/4220174572 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1028406228 0 0) 0x7f7d2c0b5d80 con 0x7f7d2c06ff90 2024-03-19T01:23:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d315ef640 1 -- 172.21.15.7:0/4220174572 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2c148860 con 0x7f7d2c149340 2024-03-19T01:23:20.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d315ef640 1 --2- 172.21.15.7:0/4220174572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 0x7f7d2c150a60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0097c0 tx=0x7f7d1c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108980cd24a874df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d2b7fe640 1 -- 172.21.15.7:0/4220174572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0b0070 con 0x7f7d2c149340 2024-03-19T01:23:20.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d2b7fe640 1 -- 172.21.15.7:0/4220174572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d1c0ab600 con 0x7f7d2c149340 2024-03-19T01:23:20.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.716+0000 7f7d2b7fe640 1 -- 172.21.15.7:0/4220174572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0ab9d0 con 0x7f7d2c149340 2024-03-19T01:23:20.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- 172.21.15.7:0/4220174572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 msgr2=0x7f7d2c150a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 --2- 172.21.15.7:0/4220174572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 0x7f7d2c150a60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0097c0 tx=0x7f7d1c01c340 comp rx=0 tx=0).stop 2024-03-19T01:23:20.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- 172.21.15.7:0/4220174572 shutdown_connections 2024-03-19T01:23:20.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 --2- 172.21.15.7:0/4220174572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 0x7f7d2c150a60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- 172.21.15.7:0/4220174572 >> 172.21.15.7:0/4220174572 conn(0x7f7d2c06c3b0 msgr2=0x7f7d2c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- 172.21.15.7:0/4220174572 shutdown_connections 2024-03-19T01:23:20.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- 172.21.15.7:0/4220174572 wait complete. 2024-03-19T01:23:20.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 Processor -- start 2024-03-19T01:23:20.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- start start 2024-03-19T01:23:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d2c06ff90 0x7f7d2c146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d2c0b59a0 0x7f7d2c147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 0x7f7d2c1e0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c0b4f50 con 0x7f7d2c149340 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c0b4dd0 con 0x7f7d2c0b59a0 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7d2c1537c0 con 0x7f7d2c06ff90 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d315ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 0x7f7d2c1e0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d315ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 0x7f7d2c1e0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42192/0 (socket says 172.21.15.7:42192) 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d315ef640 1 -- 172.21.15.7:0/1849555982 learned_addr learned my addr 172.21.15.7:0/1849555982 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d30dee640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d2c06ff90 0x7f7d2c146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d2bfff640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d2c0b59a0 0x7f7d2c147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d315ef640 1 -- 172.21.15.7:0/1849555982 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d2c06ff90 msgr2=0x7f7d2c146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d315ef640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d2c06ff90 0x7f7d2c146290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d315ef640 1 -- 172.21.15.7:0/1849555982 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d2c0b59a0 msgr2=0x7f7d2c147ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d315ef640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d2c0b59a0 0x7f7d2c147ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d315ef640 1 -- 172.21.15.7:0/1849555982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2c1e14d0 con 0x7f7d2c149340 2024-03-19T01:23:20.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d2bfff640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d2c0b59a0 0x7f7d2c147ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:20.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d315ef640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 0x7f7d2c1e0f90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0ae040 tx=0x7f7d1c01ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d297fa640 1 -- 172.21.15.7:0/1849555982 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0b0040 con 0x7f7d2c149340 2024-03-19T01:23:20.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d297fa640 1 -- 172.21.15.7:0/1849555982 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d1c004030 con 0x7f7d2c149340 2024-03-19T01:23:20.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d297fa640 1 -- 172.21.15.7:0/1849555982 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0043e0 con 0x7f7d2c149340 2024-03-19T01:23:20.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d2c146920 con 0x7f7d2c149340 2024-03-19T01:23:20.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d2c136cd0 con 0x7f7d2c149340 2024-03-19T01:23:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.720+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d2c075570 con 0x7f7d2c149340 2024-03-19T01:23:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.724+0000 7f7d297fa640 1 -- 172.21.15.7:0/1849555982 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7d1c003620 con 0x7f7d2c149340 2024-03-19T01:23:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.724+0000 7f7d297fa640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7d0c067660 0x7f7d0c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.724+0000 7f7d297fa640 1 -- 172.21.15.7:0/1849555982 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f7d1c1200a0 con 0x7f7d2c149340 2024-03-19T01:23:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.724+0000 7f7d30dee640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7d0c067660 0x7f7d0c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.724+0000 7f7d30dee640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7d0c067660 0x7f7d0c069b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7d24005530 tx=0x7f7d24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.728+0000 7f7d297fa640 1 -- 172.21.15.7:0/1849555982 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7d1c016050 con 0x7f7d2c149340 2024-03-19T01:23:20.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7d2c0b6b00 con 0x7f7d2c149340 2024-03-19T01:23:20.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d297fa640 1 -- 172.21.15.7:0/1849555982 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f7d1c0f1100 con 0x7f7d2c149340 2024-03-19T01:23:20.887 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:20.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7d0c067660 msgr2=0x7f7d0c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7d0c067660 0x7f7d0c069b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7d24005530 tx=0x7f7d24009040 comp rx=0 tx=0).stop 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 msgr2=0x7f7d2c1e0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 0x7f7d2c1e0f90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0ae040 tx=0x7f7d1c01ab80 comp rx=0 tx=0).stop 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 shutdown_connections 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7d0c067660 0x7f7d0c069b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d2c149340 0x7f7d2c1e0f90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d2c0b59a0 0x7f7d2c147ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 --2- 172.21.15.7:0/1849555982 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d2c06ff90 0x7f7d2c146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 >> 172.21.15.7:0/1849555982 conn(0x7f7d2c06c3b0 msgr2=0x7f7d2c0b2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 shutdown_connections 2024-03-19T01:23:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:20.884+0000 7f7d33052640 1 -- 172.21.15.7:0/1849555982 wait complete. 2024-03-19T01:23:20.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:21.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3f720640 1 Processor -- start 2024-03-19T01:23:21.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3f720640 1 -- start start 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3f720640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fcf38074390 0x7fcf38145e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3f720640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcf3813f5d0 0x7fcf380b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3f720640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf380711c0 con 0x7fcf380701a0 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3f720640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fcf38071430 con 0x7fcf3813f5d0 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3f720640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fcf38074770 con 0x7fcf38074390 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3ccbb640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcf380701a0 0x7fcf3813eec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52910/0 (socket says 172.21.15.7:52910) 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3dcbd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcf3813f5d0 0x7fcf380b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3ccbb640 1 -- 172.21.15.7:0/3994335451 learned_addr learned my addr 172.21.15.7:0/3994335451 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3d4bc640 1 --2- 172.21.15.7:0/3994335451 >> v2:172.21.15.167:3300/0 conn(0x7fcf38074390 0x7fcf38145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:21.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf2ffff640 1 -- 172.21.15.7:0/3994335451 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 847436068 0 0) 0x7fcf380711c0 con 0x7fcf380701a0 2024-03-19T01:23:21.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf2ffff640 1 -- 172.21.15.7:0/3994335451 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf24003530 con 0x7fcf380701a0 2024-03-19T01:23:21.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3dcbd640 1 -- 172.21.15.7:0/3994335451 >> v2:172.21.15.167:3300/0 conn(0x7fcf38074390 msgr2=0x7fcf38145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:21.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3dcbd640 1 --2- 172.21.15.7:0/3994335451 >> v2:172.21.15.167:3300/0 conn(0x7fcf38074390 0x7fcf38145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:21.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3dcbd640 1 -- 172.21.15.7:0/3994335451 >> v1:172.21.15.7:6789/0 conn(0x7fcf380701a0 legacy=0x7fcf3813eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:21.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.012+0000 7fcf3dcbd640 1 -- 172.21.15.7:0/3994335451 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf38145340 con 0x7fcf3813f5d0 2024-03-19T01:23:21.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3dcbd640 1 --2- 172.21.15.7:0/3994335451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf3813f5d0 0x7fcf380b6f30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf380ba060 tx=0x7fcf3001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898b5a2c116eed59 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:21.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf2ffff640 1 -- 172.21.15.7:0/3994335451 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf300b2070 con 0x7fcf3813f5d0 2024-03-19T01:23:21.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf2ffff640 1 -- 172.21.15.7:0/3994335451 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf300a5930 con 0x7fcf3813f5d0 2024-03-19T01:23:21.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf2ffff640 1 -- 172.21.15.7:0/3994335451 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf300a5ce0 con 0x7fcf3813f5d0 2024-03-19T01:23:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 -- 172.21.15.7:0/3994335451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf3813f5d0 msgr2=0x7fcf380b6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 --2- 172.21.15.7:0/3994335451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf3813f5d0 0x7fcf380b6f30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf380ba060 tx=0x7fcf3001c870 comp rx=0 tx=0).stop 2024-03-19T01:23:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 -- 172.21.15.7:0/3994335451 shutdown_connections 2024-03-19T01:23:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 --2- 172.21.15.7:0/3994335451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf3813f5d0 0x7fcf380b6f30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 --2- 172.21.15.7:0/3994335451 >> v2:172.21.15.167:3300/0 conn(0x7fcf38074390 0x7fcf38145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 -- 172.21.15.7:0/3994335451 >> 172.21.15.7:0/3994335451 conn(0x7fcf380b9170 msgr2=0x7fcf380b9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:21.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 -- 172.21.15.7:0/3994335451 shutdown_connections 2024-03-19T01:23:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 -- 172.21.15.7:0/3994335451 wait complete. 2024-03-19T01:23:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 Processor -- start 2024-03-19T01:23:21.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.016+0000 7fcf3f720640 1 -- start start 2024-03-19T01:23:21.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3f720640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf380701a0 0x7fcf38073240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:21.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3f720640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf38074390 0x7fcf381344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:21.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3f720640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf3813f5d0 0x7fcf3813d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:21.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3f720640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcf381523d0 con 0x7fcf38074390 2024-03-19T01:23:21.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3f720640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3806fa00 con 0x7fcf3813f5d0 2024-03-19T01:23:21.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3f720640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcf38152550 con 0x7fcf380701a0 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3ccbb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf38074390 0x7fcf381344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3ccbb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf38074390 0x7fcf381344c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42218/0 (socket says 172.21.15.7:42218) 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3ccbb640 1 -- 172.21.15.7:0/2975289358 learned_addr learned my addr 172.21.15.7:0/2975289358 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3d4bc640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf380701a0 0x7fcf38073240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3ccbb640 1 -- 172.21.15.7:0/2975289358 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf380701a0 msgr2=0x7fcf38073240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3ccbb640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf380701a0 0x7fcf38073240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3ccbb640 1 -- 172.21.15.7:0/2975289358 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf3813f5d0 msgr2=0x7fcf3813d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3dcbd640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf3813f5d0 0x7fcf3813d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3ccbb640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf3813f5d0 0x7fcf3813d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3ccbb640 1 -- 172.21.15.7:0/2975289358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf38133a20 con 0x7fcf38074390 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3ccbb640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf38074390 0x7fcf381344c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf28001ec0 tx=0x7fcf2800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf2dffb640 1 -- 172.21.15.7:0/2975289358 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf28017070 con 0x7fcf38074390 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf2dffb640 1 -- 172.21.15.7:0/2975289358 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf28005d50 con 0x7fcf38074390 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf38139990 con 0x7fcf38074390 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf2dffb640 1 -- 172.21.15.7:0/2975289358 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf28007270 con 0x7fcf38074390 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf381351f0 con 0x7fcf38074390 2024-03-19T01:23:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf38075460 con 0x7fcf38074390 2024-03-19T01:23:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf2dffb640 1 -- 172.21.15.7:0/2975289358 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcf2801a070 con 0x7fcf38074390 2024-03-19T01:23:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf2dffb640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcf1c067660 0x7fcf1c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf2dffb640 1 -- 172.21.15.7:0/2975289358 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fcf280848a0 con 0x7fcf38074390 2024-03-19T01:23:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3d4bc640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcf1c067660 0x7fcf1c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:21.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.020+0000 7fcf3d4bc640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcf1c067660 0x7fcf1c069b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009a30 tx=0x7fcf34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.024+0000 7fcf2dffb640 1 -- 172.21.15.7:0/2975289358 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcf38139990 con 0x7fcf38074390 2024-03-19T01:23:21.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.212+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf38134160 con 0x7fcf38074390 2024-03-19T01:23:21.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.212+0000 7fcf2dffb640 1 -- 172.21.15.7:0/2975289358 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7fcf280559c0 con 0x7fcf38074390 2024-03-19T01:23:21.217 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:21.217 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T01:23:21.218 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 2 2024-03-19T01:23:21.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcf1c067660 msgr2=0x7fcf1c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcf1c067660 0x7fcf1c069b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009a30 tx=0x7fcf34008040 comp rx=0 tx=0).stop 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf38074390 msgr2=0x7fcf381344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf38074390 0x7fcf381344c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf28001ec0 tx=0x7fcf2800c960 comp rx=0 tx=0).stop 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 shutdown_connections 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcf1c067660 0x7fcf1c069b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf3813f5d0 0x7fcf3813d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf38074390 0x7fcf381344c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 --2- 172.21.15.7:0/2975289358 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf380701a0 0x7fcf38073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 >> 172.21.15.7:0/2975289358 conn(0x7fcf380b9170 msgr2=0x7fcf38131ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 shutdown_connections 2024-03-19T01:23:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:21.216+0000 7fcf3f720640 1 -- 172.21.15.7:0/2975289358 wait complete. 2024-03-19T01:23:22.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:23:22.982 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:23:23.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:23.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.364+0000 7f3898636640 1 Processor -- start 2024-03-19T01:23:23.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.364+0000 7f3898636640 1 -- start start 2024-03-19T01:23:23.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f3898636640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f38900b1310 0x7f389013d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f3898636640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f38900b16f0 0x7f389013dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f3898636640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38900b2ca0 con 0x7f38900b8140 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f3898636640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f38900b7080 con 0x7f38900b1310 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f3898636640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f38900b29c0 con 0x7f38900b16f0 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38963d2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f38900b1310 0x7f389013d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38963d2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f38900b1310 0x7f389013d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42888/0 (socket says 172.21.15.7:42888) 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38963d2640 1 -- 172.21.15.7:0/3596611144 learned_addr learned my addr 172.21.15.7:0/3596611144 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38963d2640 1 -- 172.21.15.7:0/3596611144 >> v2:172.21.15.167:3300/0 conn(0x7f38900b16f0 msgr2=0x7f389013dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38963d2640 1 --2- 172.21.15.7:0/3596611144 >> v2:172.21.15.167:3300/0 conn(0x7f38900b16f0 0x7f389013dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38963d2640 1 -- 172.21.15.7:0/3596611144 >> v1:172.21.15.167:6789/0 conn(0x7f38900b8140 legacy=0x7f3890143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:23.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38963d2640 1 -- 172.21.15.7:0/3596611144 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38900b7320 con 0x7f38900b1310 2024-03-19T01:23:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38963d2640 1 --2- 172.21.15.7:0/3596611144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 0x7f389013d660 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f38900b9920 tx=0x7f388801d760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcf3c01c99e1776 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38953d0640 1 -- 172.21.15.7:0/3596611144 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38880b3070 con 0x7f38900b1310 2024-03-19T01:23:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38953d0640 1 -- 172.21.15.7:0/3596611144 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38880a6750 con 0x7f38900b1310 2024-03-19T01:23:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.368+0000 7f38953d0640 1 -- 172.21.15.7:0/3596611144 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388800b040 con 0x7f38900b1310 2024-03-19T01:23:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- 172.21.15.7:0/3596611144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 msgr2=0x7f389013d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 --2- 172.21.15.7:0/3596611144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 0x7f389013d660 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f38900b9920 tx=0x7f388801d760 comp rx=0 tx=0).stop 2024-03-19T01:23:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- 172.21.15.7:0/3596611144 shutdown_connections 2024-03-19T01:23:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 --2- 172.21.15.7:0/3596611144 >> v2:172.21.15.167:3300/0 conn(0x7f38900b16f0 0x7f389013dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 --2- 172.21.15.7:0/3596611144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 0x7f389013d660 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- 172.21.15.7:0/3596611144 >> 172.21.15.7:0/3596611144 conn(0x7f38900b5330 msgr2=0x7f38900b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- 172.21.15.7:0/3596611144 shutdown_connections 2024-03-19T01:23:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- 172.21.15.7:0/3596611144 wait complete. 2024-03-19T01:23:23.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 Processor -- start 2024-03-19T01:23:23.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- start start 2024-03-19T01:23:23.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 0x7f389006f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38900b16f0 0x7f38901e39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38900b8140 0x7f389013b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f38901504b0 con 0x7f38900b1310 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3890150330 con 0x7f38900b16f0 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3890150630 con 0x7f38900b8140 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f38963d2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 0x7f389006f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f38963d2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 0x7f389006f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42902/0 (socket says 172.21.15.7:42902) 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f38963d2640 1 -- 172.21.15.7:0/1807096103 learned_addr learned my addr 172.21.15.7:0/1807096103 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f38963d2640 1 -- 172.21.15.7:0/1807096103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38900b8140 msgr2=0x7f389013b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f38963d2640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38900b8140 0x7f389013b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f38963d2640 1 -- 172.21.15.7:0/1807096103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38900b16f0 msgr2=0x7f38901e39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f38963d2640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38900b16f0 0x7f38901e39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f38963d2640 1 -- 172.21.15.7:0/1807096103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3890071430 con 0x7f38900b1310 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f38963d2640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 0x7f389006f6c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f388800a1c0 tx=0x7f388801bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:23.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f388effd640 1 -- 172.21.15.7:0/1807096103 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38880be070 con 0x7f38900b1310 2024-03-19T01:23:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f388effd640 1 -- 172.21.15.7:0/1807096103 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38880ae850 con 0x7f38900b1310 2024-03-19T01:23:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38901e31d0 con 0x7f38900b1310 2024-03-19T01:23:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f388effd640 1 -- 172.21.15.7:0/1807096103 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38880b3070 con 0x7f38900b1310 2024-03-19T01:23:23.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38900af5c0 con 0x7f38900b1310 2024-03-19T01:23:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.372+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3890075460 con 0x7f38900b1310 2024-03-19T01:23:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.376+0000 7f388effd640 1 -- 172.21.15.7:0/1807096103 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f38880c1070 con 0x7f38900b1310 2024-03-19T01:23:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.376+0000 7f388effd640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f387c067660 0x7f387c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.376+0000 7f388effd640 1 -- 172.21.15.7:0/1807096103 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f388811f440 con 0x7f38900b1310 2024-03-19T01:23:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.376+0000 7f3895bd1640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f387c067660 0x7f387c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:23.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.376+0000 7f3895bd1640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f387c067660 0x7f387c069b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f389013b260 tx=0x7f3884006ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:23.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.380+0000 7f388effd640 1 -- 172.21.15.7:0/1807096103 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f38880174a0 con 0x7f38900b1310 2024-03-19T01:23:23.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.576+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38901411d0 con 0x7f38900b1310 2024-03-19T01:23:23.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.576+0000 7f388effd640 1 -- 172.21.15.7:0/1807096103 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2213 (secure 0 0 0) 0x7f38880f0380 con 0x7f38900b1310 2024-03-19T01:23:23.581 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:23.581 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4208,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/656985991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":656985991},{"type":"v1","addr":"172.21.15.167:6833","nonce":656985991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T01:23:23.581 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 3 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f387c067660 msgr2=0x7f387c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f387c067660 0x7f387c069b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f389013b260 tx=0x7f3884006ca0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 msgr2=0x7f389006f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 0x7f389006f6c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f388800a1c0 tx=0x7f388801bfa0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 shutdown_connections 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f387c067660 0x7f387c069b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38900b8140 0x7f389013b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38900b16f0 0x7f38901e39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 --2- 172.21.15.7:0/1807096103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38900b1310 0x7f389006f6c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 >> 172.21.15.7:0/1807096103 conn(0x7f38900b5330 msgr2=0x7f38900b8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:23.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 shutdown_connections 2024-03-19T01:23:23.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.580+0000 7f3898636640 1 -- 172.21.15.7:0/1807096103 wait complete. 2024-03-19T01:23:23.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:23.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.708+0000 7f56c52b2640 1 Processor -- start 2024-03-19T01:23:23.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.708+0000 7f56c52b2640 1 -- start start 2024-03-19T01:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56c52b2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f56c006f1c0 0x7f56c013f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56c52b2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c00b85a0 con 0x7f56c006f640 2024-03-19T01:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56c52b2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c00b20c0 con 0x7f56c00b81c0 2024-03-19T01:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56c52b2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f56c00b2710 con 0x7f56c006f1c0 2024-03-19T01:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56bf7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f56c006f640 0x7f56c014fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53088/0 (socket says 172.21.15.7:53088) 2024-03-19T01:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56bf7fe640 1 -- 172.21.15.7:0/830387987 learned_addr learned my addr 172.21.15.7:0/830387987 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56be7fc640 1 --2- 172.21.15.7:0/830387987 >> v2:172.21.15.7:3300/0 conn(0x7f56c006f1c0 0x7f56c013f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56bdffb640 1 -- 172.21.15.7:0/830387987 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928554382 0 0) 0x7f56c00b85a0 con 0x7f56c006f640 2024-03-19T01:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56bdffb640 1 -- 172.21.15.7:0/830387987 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56a8003530 con 0x7f56c006f640 2024-03-19T01:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56be7fc640 1 -- 172.21.15.7:0/830387987 >> v1:172.21.15.167:6789/0 conn(0x7f56c00b81c0 legacy=0x7f56c01486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56be7fc640 1 -- 172.21.15.7:0/830387987 >> v1:172.21.15.7:6789/0 conn(0x7f56c006f640 legacy=0x7f56c014fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.712+0000 7f56be7fc640 1 -- 172.21.15.7:0/830387987 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c00b23a0 con 0x7f56c006f1c0 2024-03-19T01:23:23.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 --2- 172.21.15.7:0/830387987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f1c0 0x7f56c013f0a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56b00097c0 tx=0x7f56b001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32795a6eb0b3e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56bdffb640 1 -- 172.21.15.7:0/830387987 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b00b0070 con 0x7f56c006f1c0 2024-03-19T01:23:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56bdffb640 1 -- 172.21.15.7:0/830387987 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b00ab600 con 0x7f56c006f1c0 2024-03-19T01:23:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56bdffb640 1 -- 172.21.15.7:0/830387987 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b00ab9d0 con 0x7f56c006f1c0 2024-03-19T01:23:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- 172.21.15.7:0/830387987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f1c0 msgr2=0x7f56c013f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 --2- 172.21.15.7:0/830387987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f1c0 0x7f56c013f0a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56b00097c0 tx=0x7f56b001c410 comp rx=0 tx=0).stop 2024-03-19T01:23:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- 172.21.15.7:0/830387987 shutdown_connections 2024-03-19T01:23:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 --2- 172.21.15.7:0/830387987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f1c0 0x7f56c013f0a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- 172.21.15.7:0/830387987 >> 172.21.15.7:0/830387987 conn(0x7f56c00b5330 msgr2=0x7f56c00b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- 172.21.15.7:0/830387987 shutdown_connections 2024-03-19T01:23:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- 172.21.15.7:0/830387987 wait complete. 2024-03-19T01:23:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 Processor -- start 2024-03-19T01:23:23.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- start start 2024-03-19T01:23:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c006f1c0 0x7f56c00b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f640 0x7f56c0072200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56c00b81c0 0x7f56c012ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f56c0153bc0 con 0x7f56c006f640 2024-03-19T01:23:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f56c0070960 con 0x7f56c00b81c0 2024-03-19T01:23:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f56c0153d40 con 0x7f56c006f1c0 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f640 0x7f56c0072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f640 0x7f56c0072200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42906/0 (socket says 172.21.15.7:42906) 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 -- 172.21.15.7:0/2329744088 learned_addr learned my addr 172.21.15.7:0/2329744088 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 -- 172.21.15.7:0/2329744088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c006f1c0 msgr2=0x7f56c00b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c006f1c0 0x7f56c00b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 -- 172.21.15.7:0/2329744088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56c00b81c0 msgr2=0x7f56c012ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56c00b81c0 0x7f56c012ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 -- 172.21.15.7:0/2329744088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c00b4490 con 0x7f56c006f640 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56be7fc640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f640 0x7f56c0072200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f56b00097c0 tx=0x7f56b00191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56a77fe640 1 -- 172.21.15.7:0/2329744088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b00b0040 con 0x7f56c006f640 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56a77fe640 1 -- 172.21.15.7:0/2329744088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b00b9070 con 0x7f56c006f640 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c00b4f30 con 0x7f56c006f640 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56a77fe640 1 -- 172.21.15.7:0/2329744088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b00b4850 con 0x7f56c006f640 2024-03-19T01:23:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.716+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c0143840 con 0x7f56c006f640 2024-03-19T01:23:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.720+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c0075460 con 0x7f56c006f640 2024-03-19T01:23:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.720+0000 7f56a77fe640 1 -- 172.21.15.7:0/2329744088 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f56b00b49f0 con 0x7f56c006f640 2024-03-19T01:23:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.720+0000 7f56a77fe640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f56ac067690 0x7f56ac069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.720+0000 7f56a77fe640 1 -- 172.21.15.7:0/2329744088 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f56b011f380 con 0x7f56c006f640 2024-03-19T01:23:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.720+0000 7f56beffd640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f56ac067690 0x7f56ac069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.720+0000 7f56beffd640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f56ac067690 0x7f56ac069b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f56b4000f70 tx=0x7f56b40079e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.724+0000 7f56a77fe640 1 -- 172.21.15.7:0/2329744088 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f56b0016090 con 0x7f56c006f640 2024-03-19T01:23:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.880+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56c0147070 con 0x7f56c006f640 2024-03-19T01:23:23.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56a77fe640 1 -- 172.21.15.7:0/2329744088 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7f56b00f0340 con 0x7f56c006f640 2024-03-19T01:23:23.888 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:23.888 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:17.421890+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f56ac067690 msgr2=0x7f56ac069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f56ac067690 0x7f56ac069b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f56b4000f70 tx=0x7f56b40079e0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f640 msgr2=0x7f56c0072200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f640 0x7f56c0072200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f56b00097c0 tx=0x7f56b00191b0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 shutdown_connections 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f56ac067690 0x7f56ac069b50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56c00b81c0 0x7f56c012ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56c006f640 0x7f56c0072200 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 --2- 172.21.15.7:0/2329744088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c006f1c0 0x7f56c00b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 >> 172.21.15.7:0/2329744088 conn(0x7f56c00b5330 msgr2=0x7f56c01464e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 shutdown_connections 2024-03-19T01:23:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:23.884+0000 7f56c52b2640 1 -- 172.21.15.7:0/2329744088 wait complete. 2024-03-19T01:23:23.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:23.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:24.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5b4c29640 1 Processor -- start 2024-03-19T01:23:24.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5b4c29640 1 -- start start 2024-03-19T01:23:24.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5b4c29640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b006ff90 0x7fd5b0071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5b4c29640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fd5b0072440 0x7fd5b014fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5b4c29640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b00b5d80 con 0x7fd5b00b59a0 2024-03-19T01:23:24.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5b4c29640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b00704c0 con 0x7fd5b0072440 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5b4c29640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b0072a80 con 0x7fd5b006ff90 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5ae59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd5b00b59a0 0x7fd5b01450d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53090/0 (socket says 172.21.15.7:53090) 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5ae59c640 1 -- 172.21.15.7:0/1399459388 learned_addr learned my addr 172.21.15.7:0/1399459388 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5a7fff640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b006ff90 0x7fd5b0071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5aed9d640 1 --2- 172.21.15.7:0/1399459388 >> v2:172.21.15.167:3300/0 conn(0x7fd5b0072440 0x7fd5b014fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97153021 0 0) 0x7fd5b00b5d80 con 0x7fd5b00b59a0 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd59c003530 con 0x7fd5b00b59a0 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4171947931 0 0) 0x7fd59c003530 con 0x7fd5b00b59a0 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b006ff90 msgr2=0x7fd5b0071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b006ff90 0x7fd5b0071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 >> v2:172.21.15.167:3300/0 conn(0x7fd5b0072440 msgr2=0x7fd5b014fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 --2- 172.21.15.7:0/1399459388 >> v2:172.21.15.167:3300/0 conn(0x7fd5b0072440 0x7fd5b014fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b0148860 con 0x7fd5b00b59a0 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fd5a0001680 con 0x7fd5b00b59a0 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 >> v1:172.21.15.7:6789/0 conn(0x7fd5b00b59a0 legacy=0x7fd5b01450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b006ff90 0x7fd59c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd59c003f40 0x7fd59c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd59c004d50 0x7fd59c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b00704c0 con 0x7fd59c004d50 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b0072a80 con 0x7fd59c003f40 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd59c00bab0 con 0x7fd5b006ff90 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5aed9d640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd59c004d50 0x7fd59c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5ae59c640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b006ff90 0x7fd59c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5a7fff640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd59c003f40 0x7fd59c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5aed9d640 1 -- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b006ff90 msgr2=0x7fd59c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5aed9d640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b006ff90 0x7fd59c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5aed9d640 1 -- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd59c003f40 msgr2=0x7fd59c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5aed9d640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd59c003f40 0x7fd59c004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.020+0000 7fd5aed9d640 1 -- 172.21.15.7:0/1399459388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59c003530 con 0x7fd59c004d50 2024-03-19T01:23:24.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5aed9d640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd59c004d50 0x7fd59c0051d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0073510 tx=0x7fd5a801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491a0569724bde94 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:24.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a80b2070 con 0x7fd59c004d50 2024-03-19T01:23:24.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a80a5b40 con 0x7fd59c004d50 2024-03-19T01:23:24.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5add9b640 1 -- 172.21.15.7:0/1399459388 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8009c10 con 0x7fd59c004d50 2024-03-19T01:23:24.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 -- 172.21.15.7:0/1399459388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd59c004d50 msgr2=0x7fd59c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 --2- 172.21.15.7:0/1399459388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd59c004d50 0x7fd59c0051d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0073510 tx=0x7fd5a801cc60 comp rx=0 tx=0).stop 2024-03-19T01:23:24.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5aed9d640 1 -- 172.21.15.7:0/1399459388 reap_dead start 2024-03-19T01:23:24.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 -- 172.21.15.7:0/1399459388 shutdown_connections 2024-03-19T01:23:24.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 -- 172.21.15.7:0/1399459388 >> 172.21.15.7:0/1399459388 conn(0x7fd5b006c3b0 msgr2=0x7fd5b006c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:24.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 -- 172.21.15.7:0/1399459388 shutdown_connections 2024-03-19T01:23:24.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 -- 172.21.15.7:0/1399459388 wait complete. 2024-03-19T01:23:24.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 Processor -- start 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 -- start start 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b00b7d00 0x7fd5b01e0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b0148260 0x7fd5b01e0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b0146290 0x7fd5b00ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b0150920 con 0x7fd5b00b7d00 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b01507a0 con 0x7fd5b0146290 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5b4c29640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b0150aa0 con 0x7fd5b0148260 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b00b7d00 0x7fd5b01e0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b00b7d00 0x7fd5b01e0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42930/0 (socket says 172.21.15.7:42930) 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5ae59c640 1 -- 172.21.15.7:0/4178433740 learned_addr learned my addr 172.21.15.7:0/4178433740 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5a7fff640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b0148260 0x7fd5b01e0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5aed9d640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b0146290 0x7fd5b00ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5ae59c640 1 -- 172.21.15.7:0/4178433740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b0148260 msgr2=0x7fd5b01e0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5ae59c640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b0148260 0x7fd5b01e0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5ae59c640 1 -- 172.21.15.7:0/4178433740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b0146290 msgr2=0x7fd5b00ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5ae59c640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b0146290 0x7fd5b00ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.024+0000 7fd5ae59c640 1 -- 172.21.15.7:0/4178433740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b00b80e0 con 0x7fd5b00b7d00 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5ae59c640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b00b7d00 0x7fd5b01e0850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0001b90 tx=0x7fd5a000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5a6ffd640 1 -- 172.21.15.7:0/4178433740 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000c8d0 con 0x7fd5b00b7d00 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5a6ffd640 1 -- 172.21.15.7:0/4178433740 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a0018070 con 0x7fd5b00b7d00 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5a6ffd640 1 -- 172.21.15.7:0/4178433740 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000ccd0 con 0x7fd5b00b7d00 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b0147b40 con 0x7fd5b00b7d00 2024-03-19T01:23:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b01450d0 con 0x7fd5b00b7d00 2024-03-19T01:23:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b0075570 con 0x7fd5b00b7d00 2024-03-19T01:23:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5a6ffd640 1 -- 172.21.15.7:0/4178433740 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd5a001d460 con 0x7fd5b00b7d00 2024-03-19T01:23:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5a6ffd640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd590067690 0x7fd590069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5a6ffd640 1 -- 172.21.15.7:0/4178433740 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fd5a0085360 con 0x7fd5b00b7d00 2024-03-19T01:23:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5a7fff640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd590067690 0x7fd590069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.028+0000 7fd5a7fff640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd590067690 0x7fd590069b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd5b013e910 tx=0x7fd598008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:24.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.032+0000 7fd5a6ffd640 1 -- 172.21.15.7:0/4178433740 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd5a00563c0 con 0x7fd5b00b7d00 2024-03-19T01:23:24.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.196+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd5b00b3860 con 0x7fd5b00b7d00 2024-03-19T01:23:24.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.196+0000 7fd5a6ffd640 1 -- 172.21.15.7:0/4178433740 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7fd5a005c0c0 con 0x7fd5b00b7d00 2024-03-19T01:23:24.201 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd590067690 msgr2=0x7fd590069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd590067690 0x7fd590069b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd5b013e910 tx=0x7fd598008040 comp rx=0 tx=0).stop 2024-03-19T01:23:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b00b7d00 msgr2=0x7fd5b01e0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b00b7d00 0x7fd5b01e0850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0001b90 tx=0x7fd5a000ece0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 shutdown_connections 2024-03-19T01:23:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd590067690 0x7fd590069b50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b0148260 0x7fd5b01e0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b0146290 0x7fd5b00ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 --2- 172.21.15.7:0/4178433740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b00b7d00 0x7fd5b01e0850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 >> 172.21.15.7:0/4178433740 conn(0x7fd5b006c3b0 msgr2=0x7fd5b0143880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 shutdown_connections 2024-03-19T01:23:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.200+0000 7fd5b4c29640 1 -- 172.21.15.7:0/4178433740 wait complete. 2024-03-19T01:23:24.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-19T01:23:24.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.332+0000 7f803147a640 1 Processor -- start 2024-03-19T01:23:24.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f803147a640 1 -- start start 2024-03-19T01:23:24.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f803147a640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f802c0b9ed0 0x7f802c1411d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f803147a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b47b0 0x7f802c14a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f803147a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f802c0ba2b0 con 0x7f802c141710 2024-03-19T01:23:24.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f803147a640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f802c0b6cc0 con 0x7f802c0b9ed0 2024-03-19T01:23:24.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f803147a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f802c0b52b0 con 0x7f802c0b47b0 2024-03-19T01:23:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f802affd640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f802c0b9ed0 0x7f802c1411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f802a7fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b47b0 0x7f802c14a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f802affd640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f802c0b9ed0 0x7f802c1411d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:52830/0 (socket says 172.21.15.7:52830) 2024-03-19T01:23:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f802affd640 1 -- 172.21.15.7:0/3975923743 learned_addr learned my addr 172.21.15.7:0/3975923743 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f802affd640 1 -- 172.21.15.7:0/3975923743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b47b0 msgr2=0x7f802c14a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f802affd640 1 --2- 172.21.15.7:0/3975923743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b47b0 0x7f802c14a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f802affd640 1 -- 172.21.15.7:0/3975923743 >> v1:172.21.15.167:6789/0 conn(0x7f802c141710 legacy=0x7f802c0b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.336+0000 7f802affd640 1 -- 172.21.15.7:0/3975923743 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c0b5590 con 0x7f802c0b9ed0 2024-03-19T01:23:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802affd640 1 --2- 172.21.15.7:0/3975923743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f802c0b9ed0 0x7f802c1411d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f802c0b6e40 tx=0x7f802401c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a66caf6fb9337ca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f8029ffb640 1 -- 172.21.15.7:0/3975923743 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80240a5760 con 0x7f802c0b9ed0 2024-03-19T01:23:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f8029ffb640 1 -- 172.21.15.7:0/3975923743 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80240a5900 con 0x7f802c0b9ed0 2024-03-19T01:23:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f8029ffb640 1 -- 172.21.15.7:0/3975923743 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80240a5cd0 con 0x7f802c0b9ed0 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- 172.21.15.7:0/3975923743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f802c0b9ed0 msgr2=0x7f802c1411d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 --2- 172.21.15.7:0/3975923743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f802c0b9ed0 0x7f802c1411d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f802c0b6e40 tx=0x7f802401c8c0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- 172.21.15.7:0/3975923743 shutdown_connections 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 --2- 172.21.15.7:0/3975923743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b47b0 0x7f802c14a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 --2- 172.21.15.7:0/3975923743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f802c0b9ed0 0x7f802c1411d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- 172.21.15.7:0/3975923743 >> 172.21.15.7:0/3975923743 conn(0x7f802c072df0 msgr2=0x7f802c073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- 172.21.15.7:0/3975923743 shutdown_connections 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- 172.21.15.7:0/3975923743 wait complete. 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 Processor -- start 2024-03-19T01:23:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- start start 2024-03-19T01:23:24.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f802c0b47b0 0x7f802c1470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b9ed0 0x7f802c138700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f802c141710 0x7f802c1e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f802c0b4d80 con 0x7f802c141710 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f802c0b4c00 con 0x7f802c0b47b0 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f802c154cd0 con 0x7f802c0b9ed0 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f802c141710 0x7f802c1e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f802c141710 0x7f802c1e34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42946/0 (socket says 172.21.15.7:42946) 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802b7fe640 1 -- 172.21.15.7:0/1113078692 learned_addr learned my addr 172.21.15.7:0/1113078692 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802a7fc640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b9ed0 0x7f802c138700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802affd640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f802c0b47b0 0x7f802c1470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802b7fe640 1 -- 172.21.15.7:0/1113078692 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b9ed0 msgr2=0x7f802c138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802b7fe640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b9ed0 0x7f802c138700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802b7fe640 1 -- 172.21.15.7:0/1113078692 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f802c0b47b0 msgr2=0x7f802c1470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802b7fe640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f802c0b47b0 0x7f802c1470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802b7fe640 1 -- 172.21.15.7:0/1113078692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c1e30e0 con 0x7f802c141710 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802a7fc640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b9ed0 0x7f802c138700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f802b7fe640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f802c141710 0x7f802c1e34f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c430 tx=0x7f801c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:24.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f80237fe640 1 -- 172.21.15.7:0/1113078692 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c005220 con 0x7f802c141710 2024-03-19T01:23:24.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f802c142de0 con 0x7f802c141710 2024-03-19T01:23:24.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f80237fe640 1 -- 172.21.15.7:0/1113078692 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f801c0053c0 con 0x7f802c141710 2024-03-19T01:23:24.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.340+0000 7f80237fe640 1 -- 172.21.15.7:0/1113078692 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c007270 con 0x7f802c141710 2024-03-19T01:23:24.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.344+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f802c142b00 con 0x7f802c141710 2024-03-19T01:23:24.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.344+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f802c13b0b0 con 0x7f802c141710 2024-03-19T01:23:24.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.344+0000 7f80237fe640 1 -- 172.21.15.7:0/1113078692 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f801c019020 con 0x7f802c141710 2024-03-19T01:23:24.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.344+0000 7f80237fe640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f800c067660 0x7f800c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.344+0000 7f80237fe640 1 -- 172.21.15.7:0/1113078692 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f801c084ce0 con 0x7f802c141710 2024-03-19T01:23:24.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.344+0000 7f802affd640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f800c067660 0x7f800c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.344+0000 7f802affd640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f800c067660 0x7f800c069b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f80240ae000 tx=0x7f80240a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.348+0000 7f80237fe640 1 -- 172.21.15.7:0/1113078692 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f802c13b0b0 con 0x7f802c141710 2024-03-19T01:23:24.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f802c075460 con 0x7f802c141710 2024-03-19T01:23:24.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f80237fe640 1 -- 172.21.15.7:0/1113078692 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f801c020020 con 0x7f802c141710 2024-03-19T01:23:24.512 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:24.512 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-03-19T01:23:24.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:23:24.513 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 1, 2024-03-19T01:23:24.513 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": ".mgr" 2024-03-19T01:23:24.513 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:23:24.513 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-03-19T01:23:24.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f800c067660 msgr2=0x7f800c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f800c067660 0x7f800c069b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f80240ae000 tx=0x7f80240a7070 comp rx=0 tx=0).stop 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f802c141710 msgr2=0x7f802c1e34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f802c141710 0x7f802c1e34f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c430 tx=0x7f801c00c8f0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 shutdown_connections 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f800c067660 0x7f800c069b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f802c0b9ed0 0x7f802c138700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f802c141710 0x7f802c1e34f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 --2- 172.21.15.7:0/1113078692 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f802c0b47b0 0x7f802c1470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.508+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 >> 172.21.15.7:0/1113078692 conn(0x7f802c072df0 msgr2=0x7f802c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.512+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 shutdown_connections 2024-03-19T01:23:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.512+0000 7f803147a640 1 -- 172.21.15.7:0/1113078692 wait complete. 2024-03-19T01:23:24.529 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-19T01:23:24.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-19T01:23:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe9876640 1 Processor -- start 2024-03-19T01:23:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe9876640 1 -- start start 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe9876640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe406c2f0 0x7f5fe40b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe9876640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fe406c6d0 con 0x7f5fe4148390 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe9876640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fe40b1580 con 0x7f5fe40b0430 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe9876640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5fe406d4e0 con 0x7f5fe406c2f0 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe37fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5fe4148390 0x7f5fe414fc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53094/0 (socket says 172.21.15.7:53094) 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe37fe640 1 -- 172.21.15.7:0/2943452509 learned_addr learned my addr 172.21.15.7:0/2943452509 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe2ffd640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe406c2f0 0x7f5fe40b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231816782 0 0) 0x7f5fe406c6d0 con 0x7f5fe4148390 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fd0003530 con 0x7f5fe4148390 2024-03-19T01:23:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3861067616 0 0) 0x7f5fd0003530 con 0x7f5fe4148390 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe406c2f0 msgr2=0x7f5fe40b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe406c2f0 0x7f5fe40b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 >> v1:172.21.15.167:6789/0 conn(0x7f5fe40b0430 legacy=0x7f5fe4147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe40b1cf0 con 0x7f5fe4148390 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f5fdc001680 con 0x7f5fe4148390 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 >> v1:172.21.15.7:6789/0 conn(0x7f5fe4148390 legacy=0x7f5fe414fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe406c2f0 0x7f5fd00039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd0003f70 0x7f5fd0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd0004d50 0x7f5fd00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe406d4e0 con 0x7f5fd0004d50 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd000bb60 con 0x7f5fd0003f70 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd000bd60 con 0x7f5fe406c2f0 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe37fe640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd0004d50 0x7f5fd00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe2ffd640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe406c2f0 0x7f5fd00039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe27fc640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd0003f70 0x7f5fd0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe37fe640 1 -- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe406c2f0 msgr2=0x7f5fd00039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe37fe640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe406c2f0 0x7f5fd00039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe37fe640 1 -- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd0003f70 msgr2=0x7f5fd0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe37fe640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd0003f70 0x7f5fd0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe37fe640 1 -- 172.21.15.7:0/2943452509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd0003530 con 0x7f5fd0004d50 2024-03-19T01:23:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.644+0000 7f5fe2ffd640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe406c2f0 0x7f5fd00039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe37fe640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd0004d50 0x7f5fd00051d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00c1c0 tx=0x7f5fdc01f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92b88eaada95498 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc007d00 con 0x7f5fd0004d50 2024-03-19T01:23:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fdc005b00 con 0x7f5fd0004d50 2024-03-19T01:23:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe1ffb640 1 -- 172.21.15.7:0/2943452509 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc007070 con 0x7f5fd0004d50 2024-03-19T01:23:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 -- 172.21.15.7:0/2943452509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd0004d50 msgr2=0x7f5fd00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 --2- 172.21.15.7:0/2943452509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fd0004d50 0x7f5fd00051d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00c1c0 tx=0x7f5fdc01f3d0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe37fe640 1 -- 172.21.15.7:0/2943452509 reap_dead start 2024-03-19T01:23:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 -- 172.21.15.7:0/2943452509 shutdown_connections 2024-03-19T01:23:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 -- 172.21.15.7:0/2943452509 >> 172.21.15.7:0/2943452509 conn(0x7f5fe40b8c40 msgr2=0x7f5fe40b9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 -- 172.21.15.7:0/2943452509 shutdown_connections 2024-03-19T01:23:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 -- 172.21.15.7:0/2943452509 wait complete. 2024-03-19T01:23:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 Processor -- start 2024-03-19T01:23:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 -- start start 2024-03-19T01:23:24.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd0004d50 0x7f5fe41da4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe41de6c0 0x7f5fe4070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fe413a7f0 0x7f5fe413ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe40b2590 con 0x7f5fe413a7f0 2024-03-19T01:23:24.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe40b22b0 con 0x7f5fd0004d50 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe9876640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5fe40b3470 con 0x7f5fe41de6c0 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fe413a7f0 0x7f5fe413ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fe413a7f0 0x7f5fe413ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42978/0 (socket says 172.21.15.7:42978) 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.648+0000 7f5fe37fe640 1 -- 172.21.15.7:0/2398879684 learned_addr learned my addr 172.21.15.7:0/2398879684 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe27fc640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe41de6c0 0x7f5fe4070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe2ffd640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd0004d50 0x7f5fe41da4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe37fe640 1 -- 172.21.15.7:0/2398879684 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe41de6c0 msgr2=0x7f5fe4070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe37fe640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe41de6c0 0x7f5fe4070bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe37fe640 1 -- 172.21.15.7:0/2398879684 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd0004d50 msgr2=0x7f5fe41da4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe37fe640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd0004d50 0x7f5fe41da4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe37fe640 1 -- 172.21.15.7:0/2398879684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe41ddbe0 con 0x7f5fe413a7f0 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe27fc640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe41de6c0 0x7f5fe4070bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe37fe640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fe413a7f0 0x7f5fe413ac70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc09bb20 tx=0x7f5fd0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fdbfff640 1 -- 172.21.15.7:0/2398879684 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc007d00 con 0x7f5fe413a7f0 2024-03-19T01:23:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fdbfff640 1 -- 172.21.15.7:0/2398879684 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fdc000e50 con 0x7f5fe413a7f0 2024-03-19T01:23:24.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe41ddec0 con 0x7f5fe413a7f0 2024-03-19T01:23:24.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fdbfff640 1 -- 172.21.15.7:0/2398879684 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc001c40 con 0x7f5fe413a7f0 2024-03-19T01:23:24.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe4071a30 con 0x7f5fe413a7f0 2024-03-19T01:23:24.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fe4075570 con 0x7f5fe413a7f0 2024-03-19T01:23:24.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fdbfff640 1 -- 172.21.15.7:0/2398879684 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5fdc001480 con 0x7f5fe413a7f0 2024-03-19T01:23:24.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fdbfff640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5fc4067660 0x7f5fc4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:24.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fdbfff640 1 -- 172.21.15.7:0/2398879684 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f5fdc121de0 con 0x7f5fe413a7f0 2024-03-19T01:23:24.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe2ffd640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5fc4067660 0x7f5fc4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:24.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.652+0000 7f5fe2ffd640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5fc4067660 0x7f5fc4069b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009ab0 tx=0x7f5fd4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:24.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.656+0000 7f5fdbfff640 1 -- 172.21.15.7:0/2398879684 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5fdc0190e0 con 0x7f5fe413a7f0 2024-03-19T01:23:24.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:24.820+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5fe40b7fe0 con 0x7f5fe413a7f0 2024-03-19T01:23:25.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fdbfff640 1 -- 172.21.15.7:0/2398879684 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f5fdc0f2e40 con 0x7f5fe413a7f0 2024-03-19T01:23:25.475 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' created 2024-03-19T01:23:25.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5fc4067660 msgr2=0x7f5fc4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:25.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5fc4067660 0x7f5fc4069b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009ab0 tx=0x7f5fd4008040 comp rx=0 tx=0).stop 2024-03-19T01:23:25.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fe413a7f0 msgr2=0x7f5fe413ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:25.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fe413a7f0 0x7f5fe413ac70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc09bb20 tx=0x7f5fd0005840 comp rx=0 tx=0).stop 2024-03-19T01:23:25.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 shutdown_connections 2024-03-19T01:23:25.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5fc4067660 0x7f5fc4069b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:25.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe41de6c0 0x7f5fe4070bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:25.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fe413a7f0 0x7f5fe413ac70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:25.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 --2- 172.21.15.7:0/2398879684 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd0004d50 0x7f5fe41da4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:25.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 >> 172.21.15.7:0/2398879684 conn(0x7f5fe40b8c40 msgr2=0x7f5fe4147210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:25.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 shutdown_connections 2024-03-19T01:23:25.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.472+0000 7f5fe9876640 1 -- 172.21.15.7:0/2398879684 wait complete. 2024-03-19T01:23:25.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-19T01:23:25.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.632+0000 7fec5b7ed640 1 Processor -- start 2024-03-19T01:23:25.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.632+0000 7fec5b7ed640 1 -- start start 2024-03-19T01:23:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fec5406ff90 0x7fec54071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec540b5d80 con 0x7fec540b59a0 2024-03-19T01:23:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec540704c0 con 0x7fec54072440 2024-03-19T01:23:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fec54072a80 con 0x7fec5406ff90 2024-03-19T01:23:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec58d88640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fec5406ff90 0x7fec54071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec59589640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fec540b59a0 0x7fec541450d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53100/0 (socket says 172.21.15.7:53100) 2024-03-19T01:23:25.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec59589640 1 -- 172.21.15.7:0/3680471963 learned_addr learned my addr 172.21.15.7:0/3680471963 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:25.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec58d88640 1 -- 172.21.15.7:0/3680471963 >> v1:172.21.15.167:6789/0 conn(0x7fec54072440 legacy=0x7fec5414fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:25.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec58d88640 1 -- 172.21.15.7:0/3680471963 >> v1:172.21.15.7:6789/0 conn(0x7fec540b59a0 legacy=0x7fec541450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:25.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec58d88640 1 -- 172.21.15.7:0/3680471963 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec54148860 con 0x7fec5406ff90 2024-03-19T01:23:25.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec58d88640 1 --2- 172.21.15.7:0/3680471963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec5406ff90 0x7fec54071f00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec500097c0 tx=0x7fec5001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1818222350a5ace server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:25.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec47fff640 1 -- 172.21.15.7:0/3680471963 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec500b0070 con 0x7fec5406ff90 2024-03-19T01:23:25.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec47fff640 1 -- 172.21.15.7:0/3680471963 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec500ab600 con 0x7fec5406ff90 2024-03-19T01:23:25.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec47fff640 1 -- 172.21.15.7:0/3680471963 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec500ab9d0 con 0x7fec5406ff90 2024-03-19T01:23:25.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 -- 172.21.15.7:0/3680471963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec5406ff90 msgr2=0x7fec54071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:25.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 --2- 172.21.15.7:0/3680471963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec5406ff90 0x7fec54071f00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec500097c0 tx=0x7fec5001c340 comp rx=0 tx=0).stop 2024-03-19T01:23:25.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 -- 172.21.15.7:0/3680471963 shutdown_connections 2024-03-19T01:23:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 --2- 172.21.15.7:0/3680471963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec5406ff90 0x7fec54071f00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 -- 172.21.15.7:0/3680471963 >> 172.21.15.7:0/3680471963 conn(0x7fec5406c3b0 msgr2=0x7fec5406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 -- 172.21.15.7:0/3680471963 shutdown_connections 2024-03-19T01:23:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.636+0000 7fec5b7ed640 1 -- 172.21.15.7:0/3680471963 wait complete. 2024-03-19T01:23:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 Processor -- start 2024-03-19T01:23:25.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 -- start start 2024-03-19T01:23:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec5406ff90 0x7fec54141af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec54072440 0x7fec540ace70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec540b59a0 0x7fec540a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fec54153b10 con 0x7fec540b59a0 2024-03-19T01:23:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fec54153990 con 0x7fec54072440 2024-03-19T01:23:25.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fec54153c90 con 0x7fec5406ff90 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59d8a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec540b59a0 0x7fec540a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59d8a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec540b59a0 0x7fec540a9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42996/0 (socket says 172.21.15.7:42996) 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59d8a640 1 -- 172.21.15.7:0/2381441117 learned_addr learned my addr 172.21.15.7:0/2381441117 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59589640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec5406ff90 0x7fec54141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec58d88640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec54072440 0x7fec540ace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59d8a640 1 -- 172.21.15.7:0/2381441117 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec5406ff90 msgr2=0x7fec54141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59d8a640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec5406ff90 0x7fec54141af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59d8a640 1 -- 172.21.15.7:0/2381441117 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec54072440 msgr2=0x7fec540ace70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59d8a640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec54072440 0x7fec540ace70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59d8a640 1 -- 172.21.15.7:0/2381441117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec540ad5c0 con 0x7fec540b59a0 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec58d88640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec54072440 0x7fec540ace70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec59d8a640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec540b59a0 0x7fec540a9d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c3f0 tx=0x7fec4c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:25.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec45ffb640 1 -- 172.21.15.7:0/2381441117 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c011790 con 0x7fec540b59a0 2024-03-19T01:23:25.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec45ffb640 1 -- 172.21.15.7:0/2381441117 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec4c017070 con 0x7fec540b59a0 2024-03-19T01:23:25.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec540abc00 con 0x7fec540b59a0 2024-03-19T01:23:25.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec45ffb640 1 -- 172.21.15.7:0/2381441117 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c011b70 con 0x7fec540b59a0 2024-03-19T01:23:25.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec54142790 con 0x7fec540b59a0 2024-03-19T01:23:25.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec54075570 con 0x7fec540b59a0 2024-03-19T01:23:25.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.640+0000 7fec45ffb640 1 -- 172.21.15.7:0/2381441117 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fec4c011d10 con 0x7fec540b59a0 2024-03-19T01:23:25.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.644+0000 7fec45ffb640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec38067690 0x7fec38069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:25.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.644+0000 7fec45ffb640 1 -- 172.21.15.7:0/2381441117 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fec4c085480 con 0x7fec540b59a0 2024-03-19T01:23:25.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.644+0000 7fec59589640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec38067690 0x7fec38069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:25.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.644+0000 7fec59589640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec38067690 0x7fec38069b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fec480012f0 tx=0x7fec48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:25.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.648+0000 7fec45ffb640 1 -- 172.21.15.7:0/2381441117 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fec540ad5c0 con 0x7fec540b59a0 2024-03-19T01:23:25.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:25.808+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fec540685b0 con 0x7fec540b59a0 2024-03-19T01:23:26.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.484+0000 7fec45ffb640 1 -- 172.21.15.7:0/2381441117 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7fec4c056370 con 0x7fec540b59a0 2024-03-19T01:23:26.488 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' created 2024-03-19T01:23:26.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec38067690 msgr2=0x7fec38069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:26.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec38067690 0x7fec38069b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fec480012f0 tx=0x7fec48009040 comp rx=0 tx=0).stop 2024-03-19T01:23:26.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec540b59a0 msgr2=0x7fec540a9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:26.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec540b59a0 0x7fec540a9d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c3f0 tx=0x7fec4c00c8b0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 shutdown_connections 2024-03-19T01:23:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec38067690 0x7fec38069b50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec540b59a0 0x7fec540a9d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec54072440 0x7fec540ace70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 --2- 172.21.15.7:0/2381441117 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec5406ff90 0x7fec54141af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 >> 172.21.15.7:0/2381441117 conn(0x7fec5406c3b0 msgr2=0x7fec540b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 shutdown_connections 2024-03-19T01:23:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.488+0000 7fec5b7ed640 1 -- 172.21.15.7:0/2381441117 wait complete. 2024-03-19T01:23:26.527 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-19T01:23:26.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77c700640 1 Processor -- start 2024-03-19T01:23:26.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77c700640 1 -- start start 2024-03-19T01:23:26.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77c700640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa7740b1600 0x7fa7740b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:26.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77c700640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fa7740b4eb0 0x7fa7741477d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77c700640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa774147d10 0x7fa77413de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77c700640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fa7740b5e30 con 0x7fa774147d10 2024-03-19T01:23:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77c700640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fa7741481a0 con 0x7fa7740b4eb0 2024-03-19T01:23:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77c700640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa7740b19e0 con 0x7fa7740b1600 2024-03-19T01:23:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77ac9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa774147d10 0x7fa77413de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77ac9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa774147d10 0x7fa77413de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43000/0 (socket says 172.21.15.7:43000) 2024-03-19T01:23:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77ac9d640 1 -- 172.21.15.7:0/2399091917 learned_addr learned my addr 172.21.15.7:0/2399091917 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa779c9b640 1 --2- 172.21.15.7:0/2399091917 >> v2:172.21.15.167:3300/0 conn(0x7fa7740b4eb0 0x7fa7741477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77ac9d640 1 -- 172.21.15.7:0/2399091917 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa7740b1600 msgr2=0x7fa7740b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77ac9d640 1 --2- 172.21.15.7:0/2399091917 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa7740b1600 0x7fa7740b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77ac9d640 1 -- 172.21.15.7:0/2399091917 >> v2:172.21.15.167:3300/0 conn(0x7fa7740b4eb0 msgr2=0x7fa7741477d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77ac9d640 1 --2- 172.21.15.7:0/2399091917 >> v2:172.21.15.167:3300/0 conn(0x7fa7740b4eb0 0x7fa7741477d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.648+0000 7fa77ac9d640 1 -- 172.21.15.7:0/2399091917 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7740b57e0 con 0x7fa774147d10 2024-03-19T01:23:26.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77ac9d640 1 --2- 172.21.15.7:0/2399091917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa774147d10 0x7fa77413de30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa77406e5e0 tx=0x7fa76c020d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b3ad2ca18e7f5c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:26.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77949a640 1 -- 172.21.15.7:0/2399091917 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c0a9a90 con 0x7fa774147d10 2024-03-19T01:23:26.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77949a640 1 -- 172.21.15.7:0/2399091917 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa76c0a9c30 con 0x7fa774147d10 2024-03-19T01:23:26.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77949a640 1 -- 172.21.15.7:0/2399091917 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c0b13f0 con 0x7fa774147d10 2024-03-19T01:23:26.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 -- 172.21.15.7:0/2399091917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa774147d10 msgr2=0x7fa77413de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:26.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 --2- 172.21.15.7:0/2399091917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa774147d10 0x7fa77413de30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa77406e5e0 tx=0x7fa76c020d80 comp rx=0 tx=0).stop 2024-03-19T01:23:26.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 -- 172.21.15.7:0/2399091917 shutdown_connections 2024-03-19T01:23:26.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 --2- 172.21.15.7:0/2399091917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa774147d10 0x7fa77413de30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 --2- 172.21.15.7:0/2399091917 >> v2:172.21.15.167:3300/0 conn(0x7fa7740b4eb0 0x7fa7741477d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 --2- 172.21.15.7:0/2399091917 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa7740b1600 0x7fa7740b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 -- 172.21.15.7:0/2399091917 >> 172.21.15.7:0/2399091917 conn(0x7fa77406c3b0 msgr2=0x7fa77406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:26.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 -- 172.21.15.7:0/2399091917 shutdown_connections 2024-03-19T01:23:26.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 -- 172.21.15.7:0/2399091917 wait complete. 2024-03-19T01:23:26.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 Processor -- start 2024-03-19T01:23:26.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 -- start start 2024-03-19T01:23:26.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7740b1600 0x7fa7740728e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:26.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7740b4eb0 0x7fa77413f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:26.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa774147d10 0x7fa77413c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:26.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7740b04d0 con 0x7fa7740b4eb0 2024-03-19T01:23:26.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa7740b23f0 con 0x7fa7740b1600 2024-03-19T01:23:26.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77c700640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa7740b0650 con 0x7fa774147d10 2024-03-19T01:23:26.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa779c9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7740b4eb0 0x7fa77413f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:26.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa779c9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7740b4eb0 0x7fa77413f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43014/0 (socket says 172.21.15.7:43014) 2024-03-19T01:23:26.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa779c9b640 1 -- 172.21.15.7:0/878480303 learned_addr learned my addr 172.21.15.7:0/878480303 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77a49c640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7740b1600 0x7fa7740728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa77ac9d640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa774147d10 0x7fa77413c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa779c9b640 1 -- 172.21.15.7:0/878480303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa774147d10 msgr2=0x7fa77413c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa779c9b640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa774147d10 0x7fa77413c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa779c9b640 1 -- 172.21.15.7:0/878480303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7740b1600 msgr2=0x7fa7740728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa779c9b640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7740b1600 0x7fa7740728e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa779c9b640 1 -- 172.21.15.7:0/878480303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa774141390 con 0x7fa7740b4eb0 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.652+0000 7fa779c9b640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7740b4eb0 0x7fa77413f940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa76400d800 tx=0x7fa76400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa76affd640 1 -- 172.21.15.7:0/878480303 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76400b6c0 con 0x7fa7740b4eb0 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa76affd640 1 -- 172.21.15.7:0/878480303 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa764019070 con 0x7fa7740b4eb0 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa774140100 con 0x7fa7740b4eb0 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa76affd640 1 -- 172.21.15.7:0/878480303 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76400ba60 con 0x7fa7740b4eb0 2024-03-19T01:23:26.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa774141670 con 0x7fa7740b4eb0 2024-03-19T01:23:26.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa774075570 con 0x7fa7740b4eb0 2024-03-19T01:23:26.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa76affd640 1 -- 172.21.15.7:0/878480303 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa76400bc00 con 0x7fa7740b4eb0 2024-03-19T01:23:26.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa76affd640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa758067660 0x7fa758069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:26.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa77a49c640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa758067660 0x7fa758069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:26.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa76affd640 1 -- 172.21.15.7:0/878480303 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fa764085c80 con 0x7fa7740b4eb0 2024-03-19T01:23:26.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.656+0000 7fa77a49c640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa758067660 0x7fa758069b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa770006fd0 tx=0x7fa770008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:26.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.660+0000 7fa76affd640 1 -- 172.21.15.7:0/878480303 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa7640569f0 con 0x7fa7740b4eb0 2024-03-19T01:23:26.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:26.864+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa77413ea20 con 0x7fa7740b4eb0 2024-03-19T01:23:27.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.484+0000 7fa76affd640 1 -- 172.21.15.7:0/878480303 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fa7640562f0 con 0x7fa7740b4eb0 2024-03-19T01:23:27.491 INFO:teuthology.orchestra.run.smithi007.stderr:new fs with metadata pool 2 and data pool 3 2024-03-19T01:23:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa758067660 msgr2=0x7fa758069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa758067660 0x7fa758069b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa770006fd0 tx=0x7fa770008040 comp rx=0 tx=0).stop 2024-03-19T01:23:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7740b4eb0 msgr2=0x7fa77413f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7740b4eb0 0x7fa77413f940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa76400d800 tx=0x7fa76400dcc0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 shutdown_connections 2024-03-19T01:23:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa758067660 0x7fa758069b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa774147d10 0x7fa77413c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7740b4eb0 0x7fa77413f940 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa76400d800 tx=0x7fa76400dcc0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 --2- 172.21.15.7:0/878480303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7740b1600 0x7fa7740728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 >> 172.21.15.7:0/878480303 conn(0x7fa77406c3b0 msgr2=0x7fa774142020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 shutdown_connections 2024-03-19T01:23:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.492+0000 7fa77c700640 1 -- 172.21.15.7:0/878480303 wait complete. 2024-03-19T01:23:27.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:27.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68c1a67640 1 Processor -- start 2024-03-19T01:23:27.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68c1a67640 1 -- start start 2024-03-19T01:23:27.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68c1a67640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f68bc13dd70 0x7f68bc143830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68c1a67640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68bc0b2ca0 con 0x7f68bc0b1310 2024-03-19T01:23:27.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68c1a67640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68bc0b7080 con 0x7f68bc0b16f0 2024-03-19T01:23:27.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68c1a67640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f68bc0b29c0 con 0x7f68bc13dd70 2024-03-19T01:23:27.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68bb7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f68bc0b1310 0x7f68bc13d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53114/0 (socket says 172.21.15.7:53114) 2024-03-19T01:23:27.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68bb7fe640 1 -- 172.21.15.7:0/3570975676 learned_addr learned my addr 172.21.15.7:0/3570975676 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:27.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68baffd640 1 --2- 172.21.15.7:0/3570975676 >> v2:172.21.15.167:3300/0 conn(0x7f68bc13dd70 0x7f68bc143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2114240060 0 0) 0x7f68bc0b2ca0 con 0x7f68bc0b1310 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68a4005740 con 0x7f68bc0b1310 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2854622616 0 0) 0x7f68a4005740 con 0x7f68bc0b1310 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 >> v2:172.21.15.167:3300/0 conn(0x7f68bc13dd70 msgr2=0x7f68bc143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 --2- 172.21.15.7:0/3570975676 >> v2:172.21.15.167:3300/0 conn(0x7f68bc13dd70 0x7f68bc143830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 >> v1:172.21.15.167:6789/0 conn(0x7f68bc0b16f0 legacy=0x7f68bc14e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc0b7320 con 0x7f68bc0b1310 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f68b00016a0 con 0x7f68bc0b1310 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 >> v1:172.21.15.7:6789/0 conn(0x7f68bc0b1310 legacy=0x7f68bc13d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68a4005bd0 0x7f68a4005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4006a20 0x7f68a4006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68a4007500 0x7f68a400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc0b29c0 con 0x7f68a4006a20 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f68a400e3d0 con 0x7f68a4005bd0 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f68a400e5d0 con 0x7f68a4007500 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68baffd640 1 --2- 172.21.15.7:0/3570975676 >> v2:172.21.15.167:3300/0 conn(0x7f68bc13dd70 0x7f68bc143830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68baffd640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4006a20 0x7f68a4006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68bb7fe640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68a4005bd0 0x7f68a4005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68bbfff640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68a4007500 0x7f68a400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68baffd640 1 -- 172.21.15.7:0/3570975676 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68a4007500 msgr2=0x7f68a400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68baffd640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68a4007500 0x7f68a400dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68baffd640 1 -- 172.21.15.7:0/3570975676 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68a4005bd0 msgr2=0x7f68a4005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68baffd640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68a4005bd0 0x7f68a4005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68baffd640 1 -- 172.21.15.7:0/3570975676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a4005740 con 0x7f68a4006a20 2024-03-19T01:23:27.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.656+0000 7f68bbfff640 1 -- 172.21.15.7:0/3570975676 reap_dead start 2024-03-19T01:23:27.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68baffd640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4006a20 0x7f68a4006ea0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68bc13e170 tx=0x7f68ac01cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb466c7d2c17faa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:27.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac009860 con 0x7f68a4006a20 2024-03-19T01:23:27.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ac004020 con 0x7f68a4006a20 2024-03-19T01:23:27.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68ba7fc640 1 -- 172.21.15.7:0/3570975676 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac004450 con 0x7f68a4006a20 2024-03-19T01:23:27.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68c1a67640 1 -- 172.21.15.7:0/3570975676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4006a20 msgr2=0x7f68a4006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68c1a67640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4006a20 0x7f68a4006ea0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68bc13e170 tx=0x7f68ac01cb60 comp rx=0 tx=0).stop 2024-03-19T01:23:27.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68c1a67640 1 -- 172.21.15.7:0/3570975676 shutdown_connections 2024-03-19T01:23:27.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68c1a67640 1 --2- 172.21.15.7:0/3570975676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4006a20 0x7f68a4006ea0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68c1a67640 1 -- 172.21.15.7:0/3570975676 >> 172.21.15.7:0/3570975676 conn(0x7f68bc0b5330 msgr2=0x7f68bc0b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:27.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68c1a67640 1 -- 172.21.15.7:0/3570975676 shutdown_connections 2024-03-19T01:23:27.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68c1a67640 1 -- 172.21.15.7:0/3570975676 wait complete. 2024-03-19T01:23:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68c1a67640 1 Processor -- start 2024-03-19T01:23:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.660+0000 7f68c1a67640 1 -- start start 2024-03-19T01:23:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68c1a67640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68a4006a20 0x7f68bc13a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68c1a67640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc13aec0 0x7f68bc1df2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68c1a67640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68bc1df7e0 0x7f68bc1dba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68c1a67640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1439b0 con 0x7f68bc13aec0 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68c1a67640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc143830 con 0x7f68a4006a20 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68c1a67640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f68bc143b30 con 0x7f68bc1df7e0 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc13aec0 0x7f68bc1df2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc13aec0 0x7f68bc1df2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43030/0 (socket says 172.21.15.7:43030) 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68baffd640 1 -- 172.21.15.7:0/2513139475 learned_addr learned my addr 172.21.15.7:0/2513139475 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68bb7fe640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68a4006a20 0x7f68bc13a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68bbfff640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68bc1df7e0 0x7f68bc1dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68baffd640 1 -- 172.21.15.7:0/2513139475 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68bc1df7e0 msgr2=0x7f68bc1dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68baffd640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68bc1df7e0 0x7f68bc1dba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68baffd640 1 -- 172.21.15.7:0/2513139475 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68a4006a20 msgr2=0x7f68bc13a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68baffd640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68a4006a20 0x7f68bc13a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68baffd640 1 -- 172.21.15.7:0/2513139475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc1473d0 con 0x7f68bc13aec0 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68bbfff640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68bc1df7e0 0x7f68bc1dba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:27.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68baffd640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc13aec0 0x7f68bc1df2a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0af040 tx=0x7f68a4007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:27.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68a3fff640 1 -- 172.21.15.7:0/2513139475 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac0ba020 con 0x7f68bc13aec0 2024-03-19T01:23:27.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68a3fff640 1 -- 172.21.15.7:0/2513139475 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ac005ba0 con 0x7f68bc13aec0 2024-03-19T01:23:27.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68bc13e9b0 con 0x7f68bc13aec0 2024-03-19T01:23:27.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68a3fff640 1 -- 172.21.15.7:0/2513139475 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac0a5820 con 0x7f68bc13aec0 2024-03-19T01:23:27.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68bc145020 con 0x7f68bc13aec0 2024-03-19T01:23:27.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68bc145e20 con 0x7f68bc13aec0 2024-03-19T01:23:27.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68a3fff640 1 -- 172.21.15.7:0/2513139475 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f68ac0bd020 con 0x7f68bc13aec0 2024-03-19T01:23:27.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68a3fff640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f68a8067690 0x7f68a8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68a3fff640 1 -- 172.21.15.7:0/2513139475 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f68ac11fad0 con 0x7f68bc13aec0 2024-03-19T01:23:27.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68bb7fe640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f68a8067690 0x7f68a8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.664+0000 7f68bb7fe640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f68a8067690 0x7f68a8069b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0b2e70 tx=0x7f68b0007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:27.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.668+0000 7f68a3fff640 1 -- 172.21.15.7:0/2513139475 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f68ac0c4030 con 0x7f68bc13aec0 2024-03-19T01:23:27.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.836+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68bc142810 con 0x7f68bc13aec0 2024-03-19T01:23:27.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.836+0000 7f68a3fff640 1 -- 172.21.15.7:0/2513139475 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16582 (secure 0 0 0) 0x7f68ac0168a0 con 0x7f68bc13aec0 2024-03-19T01:23:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:27.841 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:26.868272+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:27.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f68a8067690 msgr2=0x7f68a8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f68a8067690 0x7f68a8069b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0b2e70 tx=0x7f68b0007040 comp rx=0 tx=0).stop 2024-03-19T01:23:27.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc13aec0 msgr2=0x7f68bc1df2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc13aec0 0x7f68bc1df2a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0af040 tx=0x7f68a4007a50 comp rx=0 tx=0).stop 2024-03-19T01:23:27.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 shutdown_connections 2024-03-19T01:23:27.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f68a8067690 0x7f68a8069b50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f68bc1df7e0 0x7f68bc1dba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc13aec0 0x7f68bc1df2a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 --2- 172.21.15.7:0/2513139475 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68a4006a20 0x7f68bc13a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 >> 172.21.15.7:0/2513139475 conn(0x7f68bc0b5330 msgr2=0x7f68bc0b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:27.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 shutdown_connections 2024-03-19T01:23:27.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.840+0000 7f68c1a67640 1 -- 172.21.15.7:0/2513139475 wait complete. 2024-03-19T01:23:27.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.980+0000 7f638473e640 1 Processor -- start 2024-03-19T01:23:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.980+0000 7f638473e640 1 -- start start 2024-03-19T01:23:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f638473e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f637c0b1310 0x7f637c13d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f638473e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b16f0 0x7f637c13dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f638473e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f637c0b2ca0 con 0x7f637c0b8140 2024-03-19T01:23:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f638473e640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f637c0b7080 con 0x7f637c0b1310 2024-03-19T01:23:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f638473e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f637c0b29c0 con 0x7f637c0b16f0 2024-03-19T01:23:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f6381cd9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b16f0 0x7f637c13dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f63824da640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f637c0b1310 0x7f637c13d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f6381cd9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b16f0 0x7f637c13dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:58438/0 (socket says 172.21.15.7:58438) 2024-03-19T01:23:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f6382cdb640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f637c0b8140 0x7f637c143830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:45590/0 (socket says 172.21.15.7:45590) 2024-03-19T01:23:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f6381cd9640 1 -- 172.21.15.7:0/3716501511 learned_addr learned my addr 172.21.15.7:0/3716501511 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f6381cd9640 1 -- 172.21.15.7:0/3716501511 >> v2:172.21.15.167:3300/0 conn(0x7f637c0b1310 msgr2=0x7f637c13d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f6381cd9640 1 --2- 172.21.15.7:0/3716501511 >> v2:172.21.15.167:3300/0 conn(0x7f637c0b1310 0x7f637c13d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f6381cd9640 1 -- 172.21.15.7:0/3716501511 >> v1:172.21.15.167:6789/0 conn(0x7f637c0b8140 legacy=0x7f637c143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f6381cd9640 1 -- 172.21.15.7:0/3716501511 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c0b7320 con 0x7f637c0b16f0 2024-03-19T01:23:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.984+0000 7f63824da640 1 --2- 172.21.15.7:0/3716501511 >> v2:172.21.15.167:3300/0 conn(0x7f637c0b1310 0x7f637c13d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:27.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 --2- 172.21.15.7:0/3716501511 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b16f0 0x7f637c13dba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6378009870 tx=0x7f637801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28bf13990e43a252 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f63814d8640 1 -- 172.21.15.7:0/3716501511 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63780b0070 con 0x7f637c0b16f0 2024-03-19T01:23:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f63814d8640 1 -- 172.21.15.7:0/3716501511 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63780ab600 con 0x7f637c0b16f0 2024-03-19T01:23:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f63814d8640 1 -- 172.21.15.7:0/3716501511 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63780ab9d0 con 0x7f637c0b16f0 2024-03-19T01:23:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- 172.21.15.7:0/3716501511 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b16f0 msgr2=0x7f637c13dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 --2- 172.21.15.7:0/3716501511 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b16f0 0x7f637c13dba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6378009870 tx=0x7f637801c410 comp rx=0 tx=0).stop 2024-03-19T01:23:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- 172.21.15.7:0/3716501511 shutdown_connections 2024-03-19T01:23:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 --2- 172.21.15.7:0/3716501511 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b16f0 0x7f637c13dba0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 --2- 172.21.15.7:0/3716501511 >> v2:172.21.15.167:3300/0 conn(0x7f637c0b1310 0x7f637c13d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- 172.21.15.7:0/3716501511 >> 172.21.15.7:0/3716501511 conn(0x7f637c0b5330 msgr2=0x7f637c0b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- 172.21.15.7:0/3716501511 shutdown_connections 2024-03-19T01:23:27.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- 172.21.15.7:0/3716501511 wait complete. 2024-03-19T01:23:27.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 Processor -- start 2024-03-19T01:23:27.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- start start 2024-03-19T01:23:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f637c0b1310 0x7f637c139e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f637c0b16f0 0x7f637c1e0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b8140 0x7f637c1e3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f637c1527d0 con 0x7f637c0b16f0 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f637c152650 con 0x7f637c0b1310 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f637c152950 con 0x7f637c0b8140 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f637c0b16f0 0x7f637c1e0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f637c0b16f0 0x7f637c1e0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43036/0 (socket says 172.21.15.7:43036) 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 -- 172.21.15.7:0/790801347 learned_addr learned my addr 172.21.15.7:0/790801347 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6382cdb640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b8140 0x7f637c1e3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f63824da640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f637c0b1310 0x7f637c139e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 -- 172.21.15.7:0/790801347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b8140 msgr2=0x7f637c1e3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b8140 0x7f637c1e3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 -- 172.21.15.7:0/790801347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f637c0b1310 msgr2=0x7f637c139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f637c0b1310 0x7f637c139e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 -- 172.21.15.7:0/790801347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c0b3a40 con 0x7f637c0b16f0 2024-03-19T01:23:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f6381cd9640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f637c0b16f0 0x7f637c1e0600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f637801aac0 tx=0x7f63780191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f636effd640 1 -- 172.21.15.7:0/790801347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63780b0040 con 0x7f637c0b16f0 2024-03-19T01:23:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f636effd640 1 -- 172.21.15.7:0/790801347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63780b9070 con 0x7f637c0b16f0 2024-03-19T01:23:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637c1e1000 con 0x7f637c0b16f0 2024-03-19T01:23:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f636effd640 1 -- 172.21.15.7:0/790801347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63780b4670 con 0x7f637c0b16f0 2024-03-19T01:23:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.988+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637c13d670 con 0x7f637c0b16f0 2024-03-19T01:23:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.992+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f637c1e0b40 con 0x7f637c0b16f0 2024-03-19T01:23:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.992+0000 7f636effd640 1 -- 172.21.15.7:0/790801347 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f63780b4810 con 0x7f637c0b16f0 2024-03-19T01:23:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.992+0000 7f636effd640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6360067660 0x7f6360069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.992+0000 7f636effd640 1 -- 172.21.15.7:0/790801347 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f637811f150 con 0x7f637c0b16f0 2024-03-19T01:23:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.992+0000 7f63824da640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6360067660 0x7f6360069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.992+0000 7f63824da640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6360067660 0x7f6360069b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6370009970 tx=0x7f6370008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:28.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:27.996+0000 7f636effd640 1 -- 172.21.15.7:0/790801347 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f637c1e0b40 con 0x7f637c0b16f0 2024-03-19T01:23:28.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.160+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f637c075460 con 0x7f637c0b16f0 2024-03-19T01:23:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.160+0000 7f636effd640 1 -- 172.21.15.7:0/790801347 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16582 (secure 0 0 0) 0x7f6378016150 con 0x7f637c0b16f0 2024-03-19T01:23:28.165 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:28.165 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:26.868272+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:28.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6360067660 msgr2=0x7f6360069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:28.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6360067660 0x7f6360069b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6370009970 tx=0x7f6370008040 comp rx=0 tx=0).stop 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f637c0b16f0 msgr2=0x7f637c1e0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f637c0b16f0 0x7f637c1e0600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f637801aac0 tx=0x7f63780191b0 comp rx=0 tx=0).stop 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 shutdown_connections 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6360067660 0x7f6360069b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f637c0b8140 0x7f637c1e3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f637c0b16f0 0x7f637c1e0600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 --2- 172.21.15.7:0/790801347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f637c0b1310 0x7f637c139e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 >> 172.21.15.7:0/790801347 conn(0x7f637c0b5330 msgr2=0x7f637c0b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 shutdown_connections 2024-03-19T01:23:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.164+0000 7f638473e640 1 -- 172.21.15.7:0/790801347 wait complete. 2024-03-19T01:23:28.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-19T01:23:28.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021ce64640 1 Processor -- start 2024-03-19T01:23:28.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021ce64640 1 -- start start 2024-03-19T01:23:28.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021ce64640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02180b59a0 0x7f02181450d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021ce64640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02180b5d80 con 0x7f021806ff90 2024-03-19T01:23:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021ce64640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0218145610 con 0x7f0218149340 2024-03-19T01:23:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021ce64640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f02180704c0 con 0x7f02180b59a0 2024-03-19T01:23:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02180b59a0 0x7f02181450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02180b59a0 0x7f02181450d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43040/0 (socket says 172.21.15.7:43040) 2024-03-19T01:23:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021659c640 1 -- 172.21.15.7:0/60662018 learned_addr learned my addr 172.21.15.7:0/60662018 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021659c640 1 -- 172.21.15.7:0/60662018 >> v1:172.21.15.167:6789/0 conn(0x7f0218149340 legacy=0x7f021814fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021659c640 1 -- 172.21.15.7:0/60662018 >> v1:172.21.15.7:6789/0 conn(0x7f021806ff90 legacy=0x7f0218071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.312+0000 7f021659c640 1 -- 172.21.15.7:0/60662018 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0218072a80 con 0x7f02180b59a0 2024-03-19T01:23:28.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 --2- 172.21.15.7:0/60662018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02180b59a0 0x7f02181450d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0200009870 tx=0x7f020001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1c3357186e665f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:28.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021559a640 1 -- 172.21.15.7:0/60662018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02000b0070 con 0x7f02180b59a0 2024-03-19T01:23:28.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021559a640 1 -- 172.21.15.7:0/60662018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02000ab600 con 0x7f02180b59a0 2024-03-19T01:23:28.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021559a640 1 -- 172.21.15.7:0/60662018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02000ab9d0 con 0x7f02180b59a0 2024-03-19T01:23:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 -- 172.21.15.7:0/60662018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02180b59a0 msgr2=0x7f02181450d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 --2- 172.21.15.7:0/60662018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02180b59a0 0x7f02181450d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0200009870 tx=0x7f020001c440 comp rx=0 tx=0).stop 2024-03-19T01:23:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 -- 172.21.15.7:0/60662018 shutdown_connections 2024-03-19T01:23:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 --2- 172.21.15.7:0/60662018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02180b59a0 0x7f02181450d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 -- 172.21.15.7:0/60662018 >> 172.21.15.7:0/60662018 conn(0x7f021806c3b0 msgr2=0x7f021806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 -- 172.21.15.7:0/60662018 shutdown_connections 2024-03-19T01:23:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 -- 172.21.15.7:0/60662018 wait complete. 2024-03-19T01:23:28.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 Processor -- start 2024-03-19T01:23:28.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 -- start start 2024-03-19T01:23:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f021806ff90 0x7f02180aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02180b59a0 0x7f02181414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0218149340 0x7f0218142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f02180b4dd0 con 0x7f021806ff90 2024-03-19T01:23:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0218072610 con 0x7f02180b59a0 2024-03-19T01:23:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021ce64640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f02180b4f50 con 0x7f0218149340 2024-03-19T01:23:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f021806ff90 0x7f02180aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f021806ff90 0x7f02180aa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43052/0 (socket says 172.21.15.7:43052) 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 -- 172.21.15.7:0/262549144 learned_addr learned my addr 172.21.15.7:0/262549144 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f0216d9d640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0218149340 0x7f0218142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f0215d9b640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02180b59a0 0x7f02181414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 -- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0218149340 msgr2=0x7f0218142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0218149340 0x7f0218142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 -- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02180b59a0 msgr2=0x7f02181414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02180b59a0 0x7f02181414c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 -- 172.21.15.7:0/262549144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02180acd90 con 0x7f021806ff90 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f0216d9d640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0218149340 0x7f0218142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f0215d9b640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02180b59a0 0x7f02181414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.316+0000 7f021659c640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f021806ff90 0x7f02180aa1e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f02000ae040 tx=0x7f020001ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f020effd640 1 -- 172.21.15.7:0/262549144 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02000ba020 con 0x7f021806ff90 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f020effd640 1 -- 172.21.15.7:0/262549144 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02000b6620 con 0x7f021806ff90 2024-03-19T01:23:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02180a9660 con 0x7f021806ff90 2024-03-19T01:23:28.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f020effd640 1 -- 172.21.15.7:0/262549144 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02000b0040 con 0x7f021806ff90 2024-03-19T01:23:28.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02180aaf90 con 0x7f021806ff90 2024-03-19T01:23:28.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0218075570 con 0x7f021806ff90 2024-03-19T01:23:28.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f020effd640 1 -- 172.21.15.7:0/262549144 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f02000bd020 con 0x7f021806ff90 2024-03-19T01:23:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f020effd640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01f8067660 0x7f01f8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f020effd640 1 -- 172.21.15.7:0/262549144 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f020011fb30 con 0x7f021806ff90 2024-03-19T01:23:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f0215d9b640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01f8067660 0x7f01f8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.320+0000 7f0215d9b640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01f8067660 0x7f01f8069b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02180aa170 tx=0x7f0208009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:28.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.324+0000 7f020effd640 1 -- 172.21.15.7:0/262549144 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f02000c4030 con 0x7f021806ff90 2024-03-19T01:23:28.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:28.528+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f02180aab50 con 0x7f021806ff90 2024-03-19T01:23:29.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.528+0000 7f020effd640 1 -- 172.21.15.7:0/262549144 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f02000f6020 con 0x7f021806ff90 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01f8067660 msgr2=0x7f01f8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01f8067660 0x7f01f8069b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02180aa170 tx=0x7f0208009040 comp rx=0 tx=0).stop 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f021806ff90 msgr2=0x7f02180aa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f021806ff90 0x7f02180aa1e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f02000ae040 tx=0x7f020001ac80 comp rx=0 tx=0).stop 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 shutdown_connections 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01f8067660 0x7f01f8069b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0218149340 0x7f0218142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02180b59a0 0x7f02181414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 --2- 172.21.15.7:0/262549144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f021806ff90 0x7f02180aa1e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 >> 172.21.15.7:0/262549144 conn(0x7f021806c3b0 msgr2=0x7f0218143b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:29.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 shutdown_connections 2024-03-19T01:23:29.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.532+0000 7f021ce64640 1 -- 172.21.15.7:0/262549144 wait complete. 2024-03-19T01:23:29.561 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-19T01:23:29.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-19T01:23:29.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.692+0000 7fd2bbc2d640 1 Processor -- start 2024-03-19T01:23:29.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.692+0000 7fd2bbc2d640 1 -- start start 2024-03-19T01:23:29.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2bbc2d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b77f0 0x7fd2b40b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:29.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2bbc2d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd2b40b6e50 0x7fd2b414fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:29.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2bbc2d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b40b2c20 con 0x7fd2b40b69f0 2024-03-19T01:23:29.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2bbc2d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fd2b40b5d00 con 0x7fd2b40b6e50 2024-03-19T01:23:29.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2bbc2d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b40b5fe0 con 0x7fd2b40b77f0 2024-03-19T01:23:29.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2ba1ca640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd2b40b6e50 0x7fd2b414fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:29.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2b91c8640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd2b40b69f0 0x7fd2b40b4380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53134/0 (socket says 172.21.15.7:53134) 2024-03-19T01:23:29.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2b91c8640 1 -- 172.21.15.7:0/2135552725 learned_addr learned my addr 172.21.15.7:0/2135552725 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:29.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2b99c9640 1 --2- 172.21.15.7:0/2135552725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b77f0 0x7fd2b40b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:29.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2ba1ca640 1 -- 172.21.15.7:0/2135552725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b77f0 msgr2=0x7fd2b40b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:29.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2ba1ca640 1 --2- 172.21.15.7:0/2135552725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b77f0 0x7fd2b40b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:29.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2ba1ca640 1 -- 172.21.15.7:0/2135552725 >> v1:172.21.15.7:6789/0 conn(0x7fd2b40b69f0 legacy=0x7fd2b40b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:29.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2ba1ca640 1 -- 172.21.15.7:0/2135552725 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b40b64d0 con 0x7fd2b40b6e50 2024-03-19T01:23:29.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.696+0000 7fd2ba1ca640 1 --2- 172.21.15.7:0/2135552725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b6e50 0x7fd2b414fb60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd2b00097c0 tx=0x7fd2b001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac6b8b82df88e6c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:29.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2b89c7640 1 -- 172.21.15.7:0/2135552725 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b00b0070 con 0x7fd2b40b6e50 2024-03-19T01:23:29.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2b89c7640 1 -- 172.21.15.7:0/2135552725 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b00ab600 con 0x7fd2b40b6e50 2024-03-19T01:23:29.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2b89c7640 1 -- 172.21.15.7:0/2135552725 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b00ab9d0 con 0x7fd2b40b6e50 2024-03-19T01:23:29.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/2135552725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b6e50 msgr2=0x7fd2b414fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:29.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 --2- 172.21.15.7:0/2135552725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b6e50 0x7fd2b414fb60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd2b00097c0 tx=0x7fd2b001c410 comp rx=0 tx=0).stop 2024-03-19T01:23:29.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/2135552725 shutdown_connections 2024-03-19T01:23:29.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 --2- 172.21.15.7:0/2135552725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b6e50 0x7fd2b414fb60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:29.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 --2- 172.21.15.7:0/2135552725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b77f0 0x7fd2b40b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:29.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/2135552725 >> 172.21.15.7:0/2135552725 conn(0x7fd2b406b2c0 msgr2=0x7fd2b406b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:29.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/2135552725 shutdown_connections 2024-03-19T01:23:29.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/2135552725 wait complete. 2024-03-19T01:23:29.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 Processor -- start 2024-03-19T01:23:29.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 -- start start 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2b40b69f0 0x7fd2b413ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b6e50 0x7fd2b41e12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b77f0 0x7fd2b4137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b4152a40 con 0x7fd2b40b77f0 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41528c0 con 0x7fd2b40b69f0 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2bbc2d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b4152bc0 con 0x7fd2b40b6e50 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2ba1ca640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b77f0 0x7fd2b4137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2ba1ca640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b77f0 0x7fd2b4137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43060/0 (socket says 172.21.15.7:43060) 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.700+0000 7fd2ba1ca640 1 -- 172.21.15.7:0/3229872008 learned_addr learned my addr 172.21.15.7:0/3229872008 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2b91c8640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b6e50 0x7fd2b41e12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2b99c9640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2b40b69f0 0x7fd2b413ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2ba1ca640 1 -- 172.21.15.7:0/3229872008 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b6e50 msgr2=0x7fd2b41e12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2ba1ca640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b6e50 0x7fd2b41e12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:29.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2ba1ca640 1 -- 172.21.15.7:0/3229872008 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2b40b69f0 msgr2=0x7fd2b413ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:29.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2ba1ca640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2b40b69f0 0x7fd2b413ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:29.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2ba1ca640 1 -- 172.21.15.7:0/3229872008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b4133ff0 con 0x7fd2b40b77f0 2024-03-19T01:23:29.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2ba1ca640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b77f0 0x7fd2b4137b60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd2b001a040 tx=0x7fd2b00191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:29.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2aa7fc640 1 -- 172.21.15.7:0/3229872008 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b00b8020 con 0x7fd2b40b77f0 2024-03-19T01:23:29.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2aa7fc640 1 -- 172.21.15.7:0/3229872008 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b00b4410 con 0x7fd2b40b77f0 2024-03-19T01:23:29.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b4144690 con 0x7fd2b40b77f0 2024-03-19T01:23:29.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2aa7fc640 1 -- 172.21.15.7:0/3229872008 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b00b0040 con 0x7fd2b40b77f0 2024-03-19T01:23:29.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b41e1ca0 con 0x7fd2b40b77f0 2024-03-19T01:23:29.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2b4075570 con 0x7fd2b40b77f0 2024-03-19T01:23:29.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2aa7fc640 1 -- 172.21.15.7:0/3229872008 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd2b00bb020 con 0x7fd2b40b77f0 2024-03-19T01:23:29.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2aa7fc640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd298067660 0x7fd298069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:29.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2aa7fc640 1 -- 172.21.15.7:0/3229872008 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd2b011f390 con 0x7fd2b40b77f0 2024-03-19T01:23:29.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2b99c9640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd298067660 0x7fd298069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:29.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.704+0000 7fd2b99c9640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd298067660 0x7fd298069b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009e90 tx=0x7fd2ac00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:29.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.716+0000 7fd2aa7fc640 1 -- 172.21.15.7:0/3229872008 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd2b00c2030 con 0x7fd2b40b77f0 2024-03-19T01:23:29.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:29.912+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd2b41401e0 con 0x7fd2b40b77f0 2024-03-19T01:23:30.386 INFO:tasks.ceph.mds.a.smithi007.stderr:2024-03-19T01:23:30.380+0000 7f10c3e2d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T01:23:30.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.544+0000 7fd2aa7fc640 1 -- 172.21.15.7:0/3229872008 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7fd2b0016090 con 0x7fd2b40b77f0 2024-03-19T01:23:30.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd298067660 msgr2=0x7fd298069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:30.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd298067660 0x7fd298069b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009e90 tx=0x7fd2ac00b040 comp rx=0 tx=0).stop 2024-03-19T01:23:30.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b77f0 msgr2=0x7fd2b4137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:30.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b77f0 0x7fd2b4137b60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd2b001a040 tx=0x7fd2b00191b0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 shutdown_connections 2024-03-19T01:23:30.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd298067660 0x7fd298069b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2b40b6e50 0x7fd2b41e12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2b40b77f0 0x7fd2b4137b60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 --2- 172.21.15.7:0/3229872008 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2b40b69f0 0x7fd2b413ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 >> 172.21.15.7:0/3229872008 conn(0x7fd2b406b2c0 msgr2=0x7fd2b4142930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:30.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 shutdown_connections 2024-03-19T01:23:30.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.548+0000 7fd2bbc2d640 1 -- 172.21.15.7:0/3229872008 wait complete. 2024-03-19T01:23:30.573 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-19T01:23:30.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:30.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b4572f640 1 Processor -- start 2024-03-19T01:23:30.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b4572f640 1 -- start start 2024-03-19T01:23:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b4572f640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f6b400b5680 0x7f6b4006e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b4572f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b4006de00 con 0x7f6b400b0470 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b4572f640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b400af2f0 con 0x7f6b40072c50 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b4572f640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f6b400b5a60 con 0x7f6b400b5680 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6b400b0470 0x7f6b4006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53140/0 (socket says 172.21.15.7:53140) 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3f7fe640 1 -- 172.21.15.7:0/3937219063 learned_addr learned my addr 172.21.15.7:0/3937219063 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3effd640 1 --2- 172.21.15.7:0/3937219063 >> v2:172.21.15.167:3300/0 conn(0x7f6b400b5680 0x7f6b4006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1720827373 0 0) 0x7f6b4006de00 con 0x7f6b400b0470 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b2c003530 con 0x7f6b400b0470 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3972515301 0 0) 0x7f6b2c003530 con 0x7f6b400b0470 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 >> v2:172.21.15.167:3300/0 conn(0x7f6b400b5680 msgr2=0x7f6b4006e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 --2- 172.21.15.7:0/3937219063 >> v2:172.21.15.167:3300/0 conn(0x7f6b400b5680 0x7f6b4006e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 >> v1:172.21.15.167:6789/0 conn(0x7f6b40072c50 legacy=0x7f6b4006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b400afb50 con 0x7f6b400b0470 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f6b38001a00 con 0x7f6b400b0470 2024-03-19T01:23:30.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 >> v1:172.21.15.7:6789/0 conn(0x7f6b400b0470 legacy=0x7f6b4006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b2c0039c0 0x7f6b2c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b2c004810 0x7f6b2c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b2c0052f0 0x7f6b2c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6b400b5a60 con 0x7f6b2c0039c0 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2c00c1c0 con 0x7f6b2c004810 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6b2c00c3c0 con 0x7f6b2c0052f0 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3effd640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b2c0039c0 0x7f6b2c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b367fc640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b2c004810 0x7f6b2c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3f7fe640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b2c0052f0 0x7f6b2c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3effd640 1 -- 172.21.15.7:0/3937219063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b2c0052f0 msgr2=0x7f6b2c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3effd640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b2c0052f0 0x7f6b2c00b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3effd640 1 -- 172.21.15.7:0/3937219063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b2c004810 msgr2=0x7f6b2c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:30.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3effd640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b2c004810 0x7f6b2c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3f7fe640 1 -- 172.21.15.7:0/3937219063 reap_dead start 2024-03-19T01:23:30.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b3effd640 1 -- 172.21.15.7:0/3937219063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2c003530 con 0x7f6b2c0039c0 2024-03-19T01:23:30.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.696+0000 7f6b367fc640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b2c004810 0x7f6b2c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:30.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b3effd640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b2c0039c0 0x7f6b2c003de0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009ab0 tx=0x7f6b3001ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f1c54eb1dd9d0e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:30.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b300b2070 con 0x7f6b2c0039c0 2024-03-19T01:23:30.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b300a55f0 con 0x7f6b2c0039c0 2024-03-19T01:23:30.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b3e7fc640 1 -- 172.21.15.7:0/3937219063 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b300096b0 con 0x7f6b2c0039c0 2024-03-19T01:23:30.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b4572f640 1 -- 172.21.15.7:0/3937219063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b2c0039c0 msgr2=0x7f6b2c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:30.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b4572f640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b2c0039c0 0x7f6b2c003de0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009ab0 tx=0x7f6b3001ca20 comp rx=0 tx=0).stop 2024-03-19T01:23:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b4572f640 1 -- 172.21.15.7:0/3937219063 shutdown_connections 2024-03-19T01:23:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b4572f640 1 --2- 172.21.15.7:0/3937219063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b2c0039c0 0x7f6b2c003de0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b4572f640 1 -- 172.21.15.7:0/3937219063 >> 172.21.15.7:0/3937219063 conn(0x7f6b400b6510 msgr2=0x7f6b400b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b4572f640 1 -- 172.21.15.7:0/3937219063 shutdown_connections 2024-03-19T01:23:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.700+0000 7f6b4572f640 1 -- 172.21.15.7:0/3937219063 wait complete. 2024-03-19T01:23:30.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 Processor -- start 2024-03-19T01:23:30.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 -- start start 2024-03-19T01:23:30.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b2c0039c0 0x7f6b40146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:30.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b401e0be0 0x7f6b4006b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:30.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4006b800 0x7f6b4006bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:30.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6b40147a40 con 0x7f6b4006b800 2024-03-19T01:23:30.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6b401478c0 con 0x7f6b401e0be0 2024-03-19T01:23:30.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6b4006ec70 con 0x7f6b2c0039c0 2024-03-19T01:23:30.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4006b800 0x7f6b4006bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:30.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4006b800 0x7f6b4006bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43088/0 (socket says 172.21.15.7:43088) 2024-03-19T01:23:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3f7fe640 1 -- 172.21.15.7:0/2827080609 learned_addr learned my addr 172.21.15.7:0/2827080609 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b367fc640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b401e0be0 0x7f6b4006b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3effd640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b2c0039c0 0x7f6b40146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3f7fe640 1 -- 172.21.15.7:0/2827080609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b2c0039c0 msgr2=0x7f6b40146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3f7fe640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b2c0039c0 0x7f6b40146f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3f7fe640 1 -- 172.21.15.7:0/2827080609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b401e0be0 msgr2=0x7f6b4006b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3f7fe640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b401e0be0 0x7f6b4006b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3f7fe640 1 -- 172.21.15.7:0/2827080609 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b400b41c0 con 0x7f6b4006b800 2024-03-19T01:23:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b3f7fe640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4006b800 0x7f6b4006bc80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6b38001590 tx=0x7f6b3800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b37fff640 1 -- 172.21.15.7:0/2827080609 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38004d70 con 0x7f6b4006b800 2024-03-19T01:23:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b37fff640 1 -- 172.21.15.7:0/2827080609 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b3800bb20 con 0x7f6b4006b800 2024-03-19T01:23:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b401d05f0 con 0x7f6b4006b800 2024-03-19T01:23:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b37fff640 1 -- 172.21.15.7:0/2827080609 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38007110 con 0x7f6b4006b800 2024-03-19T01:23:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.704+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4013de50 con 0x7f6b4006b800 2024-03-19T01:23:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.708+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b40075460 con 0x7f6b4006b800 2024-03-19T01:23:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.708+0000 7f6b37fff640 1 -- 172.21.15.7:0/2827080609 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6b3801b020 con 0x7f6b4006b800 2024-03-19T01:23:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.708+0000 7f6b37fff640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b20067660 0x7f6b20069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.708+0000 7f6b37fff640 1 -- 172.21.15.7:0/2827080609 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6b38085980 con 0x7f6b4006b800 2024-03-19T01:23:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.708+0000 7f6b3effd640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b20067660 0x7f6b20069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.708+0000 7f6b3effd640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b20067660 0x7f6b20069b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6b300af000 tx=0x7f6b30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:30.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:30.712+0000 7f6b37fff640 1 -- 172.21.15.7:0/2827080609 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6b38056690 con 0x7f6b4006b800 2024-03-19T01:23:31.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.036+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b40144100 con 0x7f6b4006b800 2024-03-19T01:23:31.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.036+0000 7f6b37fff640 1 -- 172.21.15.7:0/2827080609 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3524 (secure 0 0 0) 0x7f6b38001980 con 0x7f6b4006b800 2024-03-19T01:23:31.041 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:31.041 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4208,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/656985991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":656985991},{"type":"v1","addr":"172.21.15.167:6833","nonce":656985991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:23:26.868603+0000","modified":"2024-03-19T01:23:29.918429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-03-19T01:23:31.041 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-03-19T01:23:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b20067660 msgr2=0x7f6b20069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b20067660 0x7f6b20069b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6b300af000 tx=0x7f6b30008040 comp rx=0 tx=0).stop 2024-03-19T01:23:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4006b800 msgr2=0x7f6b4006bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4006b800 0x7f6b4006bc80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6b38001590 tx=0x7f6b3800b4a0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 shutdown_connections 2024-03-19T01:23:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b20067660 0x7f6b20069b20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b4006b800 0x7f6b4006bc80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b401e0be0 0x7f6b4006b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 --2- 172.21.15.7:0/2827080609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b2c0039c0 0x7f6b40146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 >> 172.21.15.7:0/2827080609 conn(0x7f6b400b6510 msgr2=0x7f6b400b4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 shutdown_connections 2024-03-19T01:23:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.040+0000 7f6b4572f640 1 -- 172.21.15.7:0/2827080609 wait complete. 2024-03-19T01:23:31.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:31.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44f4a5b640 1 Processor -- start 2024-03-19T01:23:31.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44f4a5b640 1 -- start start 2024-03-19T01:23:31.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44f4a5b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f44f006bd10 0x7f44f00af380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44f4a5b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006eb60 0x7f44f006e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44f4a5b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44f00b5420 0x7f44f00b5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44f4a5b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f44f006e650 con 0x7f44f00b5420 2024-03-19T01:23:31.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44f4a5b640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f44f006dc80 con 0x7f44f006bd10 2024-03-19T01:23:31.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44f4a5b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f44f006c0f0 con 0x7f44f006eb60 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44eed9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44f00b5420 0x7f44f00b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44eed9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44f00b5420 0x7f44f00b5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43102/0 (socket says 172.21.15.7:43102) 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44eed9d640 1 -- 172.21.15.7:0/4291852926 learned_addr learned my addr 172.21.15.7:0/4291852926 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44edd9b640 1 --2- 172.21.15.7:0/4291852926 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006eb60 0x7f44f006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44ee59c640 1 --2- 172.21.15.7:0/4291852926 >> v2:172.21.15.167:3300/0 conn(0x7f44f006bd10 0x7f44f00af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44eed9d640 1 -- 172.21.15.7:0/4291852926 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006eb60 msgr2=0x7f44f006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44eed9d640 1 --2- 172.21.15.7:0/4291852926 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006eb60 0x7f44f006e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44eed9d640 1 -- 172.21.15.7:0/4291852926 >> v2:172.21.15.167:3300/0 conn(0x7f44f006bd10 msgr2=0x7f44f00af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44eed9d640 1 --2- 172.21.15.7:0/4291852926 >> v2:172.21.15.167:3300/0 conn(0x7f44f006bd10 0x7f44f00af380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44eed9d640 1 -- 172.21.15.7:0/4291852926 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f00aec10 con 0x7f44f00b5420 2024-03-19T01:23:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.172+0000 7f44ee59c640 1 --2- 172.21.15.7:0/4291852926 >> v2:172.21.15.167:3300/0 conn(0x7f44f006bd10 0x7f44f00af380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:31.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44eed9d640 1 --2- 172.21.15.7:0/4291852926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 0x7f44f00b5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f44f00b7430 tx=0x7f44e801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e682db8d984ac62 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:31.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44ed59a640 1 -- 172.21.15.7:0/4291852926 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e80ab620 con 0x7f44f00b5420 2024-03-19T01:23:31.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44ed59a640 1 -- 172.21.15.7:0/4291852926 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e80b0070 con 0x7f44f00b5420 2024-03-19T01:23:31.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44ed59a640 1 -- 172.21.15.7:0/4291852926 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e80ab9e0 con 0x7f44f00b5420 2024-03-19T01:23:31.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 -- 172.21.15.7:0/4291852926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 msgr2=0x7f44f00b5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/4291852926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 0x7f44f00b5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f44f00b7430 tx=0x7f44e801c460 comp rx=0 tx=0).stop 2024-03-19T01:23:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 -- 172.21.15.7:0/4291852926 shutdown_connections 2024-03-19T01:23:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/4291852926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 0x7f44f00b5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/4291852926 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006eb60 0x7f44f006e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/4291852926 >> v2:172.21.15.167:3300/0 conn(0x7f44f006bd10 0x7f44f00af380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 -- 172.21.15.7:0/4291852926 >> 172.21.15.7:0/4291852926 conn(0x7f44f00b6510 msgr2=0x7f44f00b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 -- 172.21.15.7:0/4291852926 shutdown_connections 2024-03-19T01:23:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 -- 172.21.15.7:0/4291852926 wait complete. 2024-03-19T01:23:31.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 Processor -- start 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 -- start start 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006bd10 0x7f44f013f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 0x7f44f0145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f00b11e0 0x7f44f013d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0072df0 con 0x7f44f00b5420 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0072c70 con 0x7f44f00b11e0 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44f4a5b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f44f0148060 con 0x7f44f006bd10 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44edd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 0x7f44f0145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44edd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 0x7f44f0145560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43118/0 (socket says 172.21.15.7:43118) 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44edd9b640 1 -- 172.21.15.7:0/3500068218 learned_addr learned my addr 172.21.15.7:0/3500068218 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44ee59c640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006bd10 0x7f44f013f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44eed9d640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f00b11e0 0x7f44f013d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44edd9b640 1 -- 172.21.15.7:0/3500068218 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006bd10 msgr2=0x7f44f013f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44edd9b640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006bd10 0x7f44f013f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44edd9b640 1 -- 172.21.15.7:0/3500068218 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f00b11e0 msgr2=0x7f44f013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44edd9b640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f00b11e0 0x7f44f013d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44edd9b640 1 -- 172.21.15.7:0/3500068218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f00b1710 con 0x7f44f00b5420 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44ee59c640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006bd10 0x7f44f013f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.176+0000 7f44edd9b640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 0x7f44f0145560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f44e000d830 tx=0x7f44e000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44e6ffd640 1 -- 172.21.15.7:0/3500068218 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e000b6c0 con 0x7f44f00b5420 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44e6ffd640 1 -- 172.21.15.7:0/3500068218 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e0019070 con 0x7f44f00b5420 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f013e410 con 0x7f44f00b5420 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44e6ffd640 1 -- 172.21.15.7:0/3500068218 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e000ba60 con 0x7f44f00b5420 2024-03-19T01:23:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f0146a00 con 0x7f44f00b5420 2024-03-19T01:23:31.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f0075460 con 0x7f44f00b5420 2024-03-19T01:23:31.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44e6ffd640 1 -- 172.21.15.7:0/3500068218 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f44e000bc00 con 0x7f44f00b5420 2024-03-19T01:23:31.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44e6ffd640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d0067660 0x7f44d0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44ee59c640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d0067660 0x7f44d0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44e6ffd640 1 -- 172.21.15.7:0/3500068218 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f44e0085bf0 con 0x7f44f00b5420 2024-03-19T01:23:31.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.180+0000 7f44ee59c640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d0067660 0x7f44d0069b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f44d800e550 tx=0x7f44d8007ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:31.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.184+0000 7f44e6ffd640 1 -- 172.21.15.7:0/3500068218 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f44e0056900 con 0x7f44f00b5420 2024-03-19T01:23:31.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.440+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44f0144ea0 con 0x7f44f00b5420 2024-03-19T01:23:31.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.440+0000 7f44e6ffd640 1 -- 172.21.15.7:0/3500068218 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7f44e0056200 con 0x7f44f00b5420 2024-03-19T01:23:31.445 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:31.445 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:31.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d0067660 msgr2=0x7f44d0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d0067660 0x7f44d0069b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f44d800e550 tx=0x7f44d8007ab0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 msgr2=0x7f44f0145560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 0x7f44f0145560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f44e000d830 tx=0x7f44e000dcf0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 shutdown_connections 2024-03-19T01:23:31.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d0067660 0x7f44d0069b20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f00b11e0 0x7f44f013d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f00b5420 0x7f44f0145560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 --2- 172.21.15.7:0/3500068218 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f006bd10 0x7f44f013f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 >> 172.21.15.7:0/3500068218 conn(0x7f44f00b6510 msgr2=0x7f44f00b4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:31.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 shutdown_connections 2024-03-19T01:23:31.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.444+0000 7f44f4a5b640 1 -- 172.21.15.7:0/3500068218 wait complete. 2024-03-19T01:23:31.464 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-19T01:23:31.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:23:31.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f771e58e640 1 Processor -- start 2024-03-19T01:23:31.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f771e58e640 1 -- start start 2024-03-19T01:23:31.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f771e58e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f77180af200 0x7f771813ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f771e58e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f771813f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f771e58e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7718072b90 con 0x7f77180727b0 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f771e58e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f7718149420 con 0x7f77180af200 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f771e58e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f771806d930 con 0x7f77180aed40 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f77177fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f77180af200 0x7f771813ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f77177fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f77180af200 0x7f771813ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43126/0 (socket says 172.21.15.7:43126) 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f77177fe640 1 -- 172.21.15.7:0/1836937432 learned_addr learned my addr 172.21.15.7:0/1836937432 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f771cb2b640 1 --2- 172.21.15.7:0/1836937432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f771813f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f77177fe640 1 -- 172.21.15.7:0/1836937432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 msgr2=0x7f771813f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f77177fe640 1 --2- 172.21.15.7:0/1836937432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f771813f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f77177fe640 1 -- 172.21.15.7:0/1836937432 >> v1:172.21.15.167:6789/0 conn(0x7f77180727b0 legacy=0x7f7718145340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f77177fe640 1 -- 172.21.15.7:0/1836937432 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7718148dd0 con 0x7f77180af200 2024-03-19T01:23:31.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f771cb2b640 1 --2- 172.21.15.7:0/1836937432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f771813f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:31.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.588+0000 7f77177fe640 1 --2- 172.21.15.7:0/1836937432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180af200 0x7f771813ef50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f770c009870 tx=0x7f770c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa3bc1aae96f562 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:31.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7716ffd640 1 -- 172.21.15.7:0/1836937432 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c0b0070 con 0x7f77180af200 2024-03-19T01:23:31.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7716ffd640 1 -- 172.21.15.7:0/1836937432 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770c0ab600 con 0x7f77180af200 2024-03-19T01:23:31.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7716ffd640 1 -- 172.21.15.7:0/1836937432 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c0ab9d0 con 0x7f77180af200 2024-03-19T01:23:31.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- 172.21.15.7:0/1836937432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180af200 msgr2=0x7f771813ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 --2- 172.21.15.7:0/1836937432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180af200 0x7f771813ef50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f770c009870 tx=0x7f770c01c520 comp rx=0 tx=0).stop 2024-03-19T01:23:31.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- 172.21.15.7:0/1836937432 shutdown_connections 2024-03-19T01:23:31.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 --2- 172.21.15.7:0/1836937432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f771813f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 --2- 172.21.15.7:0/1836937432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180af200 0x7f771813ef50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- 172.21.15.7:0/1836937432 >> 172.21.15.7:0/1836937432 conn(0x7f771806b2c0 msgr2=0x7f771806b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:31.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- 172.21.15.7:0/1836937432 shutdown_connections 2024-03-19T01:23:31.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- 172.21.15.7:0/1836937432 wait complete. 2024-03-19T01:23:31.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 Processor -- start 2024-03-19T01:23:31.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- start start 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180727b0 0x7f771809f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f7718071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77180af200 0x7f7718146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7718152680 con 0x7f77180727b0 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7718152500 con 0x7f77180af200 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7718152800 con 0x7f77180aed40 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7717fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180727b0 0x7f771809f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7717fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180727b0 0x7f771809f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43134/0 (socket says 172.21.15.7:43134) 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7717fff640 1 -- 172.21.15.7:0/3537477442 learned_addr learned my addr 172.21.15.7:0/3537477442 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771cb2b640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77180af200 0x7f7718146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f77177fe640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f7718071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7717fff640 1 -- 172.21.15.7:0/3537477442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 msgr2=0x7f7718071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7717fff640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f7718071e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7717fff640 1 -- 172.21.15.7:0/3537477442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77180af200 msgr2=0x7f7718146910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7717fff640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77180af200 0x7f7718146910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7717fff640 1 -- 172.21.15.7:0/3537477442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7718072360 con 0x7f77180727b0 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f77177fe640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f7718071e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7717fff640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180727b0 0x7f771809f0d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f771000c7c0 tx=0x7f771000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7714ff9640 1 -- 172.21.15.7:0/3537477442 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710016020 con 0x7f77180727b0 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7714ff9640 1 -- 172.21.15.7:0/3537477442 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f771000a9f0 con 0x7f77180727b0 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7718147950 con 0x7f77180727b0 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f7714ff9640 1 -- 172.21.15.7:0/3537477442 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771000ae20 con 0x7f77180727b0 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.592+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7718148430 con 0x7f77180727b0 2024-03-19T01:23:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.596+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7718075460 con 0x7f77180727b0 2024-03-19T01:23:31.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.596+0000 7f7714ff9640 1 -- 172.21.15.7:0/3537477442 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7710019020 con 0x7f77180727b0 2024-03-19T01:23:31.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.596+0000 7f7714ff9640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7704067660 0x7f7704069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.596+0000 7f77177fe640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7704067660 0x7f7704069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.596+0000 7f7714ff9640 1 -- 172.21.15.7:0/3537477442 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7710085890 con 0x7f77180727b0 2024-03-19T01:23:31.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.596+0000 7f77177fe640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7704067660 0x7f7704069b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f770c0ad000 tx=0x7f770c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:31.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.600+0000 7f7714ff9640 1 -- 172.21.15.7:0/3537477442 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7710020030 con 0x7f77180727b0 2024-03-19T01:23:31.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.756+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7718144420 con 0x7f7704067660 2024-03-19T01:23:31.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.768+0000 7f7714ff9640 1 -- 172.21.15.7:0/3537477442 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251922 (secure 0 0 0) 0x7f7718144420 con 0x7f7704067660 2024-03-19T01:23:31.772 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:23:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7704067660 msgr2=0x7f7704069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7704067660 0x7f7704069b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f770c0ad000 tx=0x7f770c008040 comp rx=0 tx=0).stop 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180727b0 msgr2=0x7f771809f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180727b0 0x7f771809f0d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f771000c7c0 tx=0x7f771000cc80 comp rx=0 tx=0).stop 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 shutdown_connections 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7704067660 0x7f7704069b20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77180aed40 0x7f7718071e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77180af200 0x7f7718146910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 --2- 172.21.15.7:0/3537477442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77180727b0 0x7f771809f0d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 >> 172.21.15.7:0/3537477442 conn(0x7f771806b2c0 msgr2=0x7f77180b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 shutdown_connections 2024-03-19T01:23:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.772+0000 7f771e58e640 1 -- 172.21.15.7:0/3537477442 wait complete. 2024-03-19T01:23:31.790 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-19T01:23:30.890859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235652,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739068,"statfs":{"total":773094113280,"available":772852805632,"internally_reserved":0,"allocated":2891776,"data_stored":1739996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.413785"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.559151+0000","last_change":"2024-03-19T01:23:27.551581+0000","last_active":"2024-03-19T01:23:27.559151+0000","last_peered":"2024-03-19T01:23:27.559151+0000","last_clean":"2024-03-19T01:23:27.559151+0000","last_became_active":"2024-03-19T01:23:27.550909+0000","last_became_peered":"2024-03-19T01:23:27.550909+0000","last_unstale":"2024-03-19T01:23:27.559151+0000","last_undegraded":"2024-03-19T01:23:27.559151+0000","last_fullsized":"2024-03-19T01:23:27.559151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498885+0000","last_change":"2024-03-19T01:23:27.512582+0000","last_active":"2024-03-19T01:23:28.498885+0000","last_peered":"2024-03-19T01:23:28.498885+0000","last_clean":"2024-03-19T01:23:28.498885+0000","last_became_active":"2024-03-19T01:23:27.512252+0000","last_became_peered":"2024-03-19T01:23:27.512252+0000","last_unstale":"2024-03-19T01:23:28.498885+0000","last_undegraded":"2024-03-19T01:23:28.498885+0000","last_fullsized":"2024-03-19T01:23:28.498885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.514378+0000","last_change":"2024-03-19T01:23:27.512676+0000","last_active":"2024-03-19T01:23:27.514378+0000","last_peered":"2024-03-19T01:23:27.514378+0000","last_clean":"2024-03-19T01:23:27.514378+0000","last_became_active":"2024-03-19T01:23:27.512311+0000","last_became_peered":"2024-03-19T01:23:27.512311+0000","last_unstale":"2024-03-19T01:23:27.514378+0000","last_undegraded":"2024-03-19T01:23:27.514378+0000","last_fullsized":"2024-03-19T01:23:27.514378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:31.282958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498730+0000","last_change":"2024-03-19T01:23:27.538857+0000","last_active":"2024-03-19T01:23:28.498730+0000","last_peered":"2024-03-19T01:23:28.498730+0000","last_clean":"2024-03-19T01:23:28.498730+0000","last_became_active":"2024-03-19T01:23:27.536808+0000","last_became_peered":"2024-03-19T01:23:27.536808+0000","last_unstale":"2024-03-19T01:23:28.498730+0000","last_undegraded":"2024-03-19T01:23:28.498730+0000","last_fullsized":"2024-03-19T01:23:28.498730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556403+0000","last_change":"2024-03-19T01:23:27.543517+0000","last_active":"2024-03-19T01:23:27.556403+0000","last_peered":"2024-03-19T01:23:27.556403+0000","last_clean":"2024-03-19T01:23:27.556403+0000","last_became_active":"2024-03-19T01:23:27.542861+0000","last_became_peered":"2024-03-19T01:23:27.542861+0000","last_unstale":"2024-03-19T01:23:27.556403+0000","last_undegraded":"2024-03-19T01:23:27.556403+0000","last_fullsized":"2024-03-19T01:23:27.556403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:12:50.425783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498639+0000","last_change":"2024-03-19T01:23:27.549333+0000","last_active":"2024-03-19T01:23:28.498639+0000","last_peered":"2024-03-19T01:23:28.498639+0000","last_clean":"2024-03-19T01:23:28.498639+0000","last_became_active":"2024-03-19T01:23:27.548338+0000","last_became_peered":"2024-03-19T01:23:27.548338+0000","last_unstale":"2024-03-19T01:23:28.498639+0000","last_undegraded":"2024-03-19T01:23:28.498639+0000","last_fullsized":"2024-03-19T01:23:28.498639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556500+0000","last_change":"2024-03-19T01:23:27.536206+0000","last_active":"2024-03-19T01:23:27.556500+0000","last_peered":"2024-03-19T01:23:27.556500+0000","last_clean":"2024-03-19T01:23:27.556500+0000","last_became_active":"2024-03-19T01:23:27.535402+0000","last_became_peered":"2024-03-19T01:23:27.535402+0000","last_unstale":"2024-03-19T01:23:27.556500+0000","last_undegraded":"2024-03-19T01:23:27.556500+0000","last_fullsized":"2024-03-19T01:23:27.556500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:10.543122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569176+0000","last_change":"2024-03-19T01:23:27.548444+0000","last_active":"2024-03-19T01:23:28.569176+0000","last_peered":"2024-03-19T01:23:28.569176+0000","last_clean":"2024-03-19T01:23:28.569176+0000","last_became_active":"2024-03-19T01:23:27.548160+0000","last_became_peered":"2024-03-19T01:23:27.548160+0000","last_unstale":"2024-03-19T01:23:28.569176+0000","last_undegraded":"2024-03-19T01:23:28.569176+0000","last_fullsized":"2024-03-19T01:23:28.569176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.560428+0000","last_change":"2024-03-19T01:23:27.558139+0000","last_active":"2024-03-19T01:23:27.560428+0000","last_peered":"2024-03-19T01:23:27.560428+0000","last_clean":"2024-03-19T01:23:27.560428+0000","last_became_active":"2024-03-19T01:23:27.557564+0000","last_became_peered":"2024-03-19T01:23:27.557564+0000","last_unstale":"2024-03-19T01:23:27.560428+0000","last_undegraded":"2024-03-19T01:23:27.560428+0000","last_fullsized":"2024-03-19T01:23:27.560428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.552973+0000","last_change":"2024-03-19T01:23:27.549847+0000","last_active":"2024-03-19T01:23:27.552973+0000","last_peered":"2024-03-19T01:23:27.552973+0000","last_clean":"2024-03-19T01:23:27.552973+0000","last_became_active":"2024-03-19T01:23:27.548601+0000","last_became_peered":"2024-03-19T01:23:27.548601+0000","last_unstale":"2024-03-19T01:23:27.552973+0000","last_undegraded":"2024-03-19T01:23:27.552973+0000","last_fullsized":"2024-03-19T01:23:27.552973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:23.802985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.560073+0000","last_change":"2024-03-19T01:23:27.558581+0000","last_active":"2024-03-19T01:23:27.560073+0000","last_peered":"2024-03-19T01:23:27.560073+0000","last_clean":"2024-03-19T01:23:27.560073+0000","last_became_active":"2024-03-19T01:23:27.558044+0000","last_became_peered":"2024-03-19T01:23:27.558044+0000","last_unstale":"2024-03-19T01:23:27.560073+0000","last_undegraded":"2024-03-19T01:23:27.560073+0000","last_fullsized":"2024-03-19T01:23:27.560073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:10.543122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831030+0000","last_change":"2024-03-19T01:23:27.543968+0000","last_active":"2024-03-19T01:23:28.831030+0000","last_peered":"2024-03-19T01:23:28.831030+0000","last_clean":"2024-03-19T01:23:28.831030+0000","last_became_active":"2024-03-19T01:23:27.543453+0000","last_became_peered":"2024-03-19T01:23:27.543453+0000","last_unstale":"2024-03-19T01:23:28.831030+0000","last_undegraded":"2024-03-19T01:23:28.831030+0000","last_fullsized":"2024-03-19T01:23:28.831030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569582+0000","last_change":"2024-03-19T01:23:27.541445+0000","last_active":"2024-03-19T01:23:28.569582+0000","last_peered":"2024-03-19T01:23:28.569582+0000","last_clean":"2024-03-19T01:23:28.569582+0000","last_became_active":"2024-03-19T01:23:27.541162+0000","last_became_peered":"2024-03-19T01:23:27.541162+0000","last_unstale":"2024-03-19T01:23:28.569582+0000","last_undegraded":"2024-03-19T01:23:28.569582+0000","last_fullsized":"2024-03-19T01:23:28.569582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.516422+0000","last_change":"2024-03-19T01:23:27.515384+0000","last_active":"2024-03-19T01:23:27.516422+0000","last_peered":"2024-03-19T01:23:27.516422+0000","last_clean":"2024-03-19T01:23:27.516422+0000","last_became_active":"2024-03-19T01:23:27.514935+0000","last_became_peered":"2024-03-19T01:23:27.514935+0000","last_unstale":"2024-03-19T01:23:27.516422+0000","last_undegraded":"2024-03-19T01:23:27.516422+0000","last_fullsized":"2024-03-19T01:23:27.516422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:54.910728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497668+0000","last_change":"2024-03-19T01:23:27.509553+0000","last_active":"2024-03-19T01:23:28.497668+0000","last_peered":"2024-03-19T01:23:28.497668+0000","last_clean":"2024-03-19T01:23:28.497668+0000","last_became_active":"2024-03-19T01:23:27.509178+0000","last_became_peered":"2024-03-19T01:23:27.509178+0000","last_unstale":"2024-03-19T01:23:28.497668+0000","last_undegraded":"2024-03-19T01:23:28.497668+0000","last_fullsized":"2024-03-19T01:23:28.497668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498214+0000","last_change":"2024-03-19T01:23:27.547056+0000","last_active":"2024-03-19T01:23:28.498214+0000","last_peered":"2024-03-19T01:23:28.498214+0000","last_clean":"2024-03-19T01:23:28.498214+0000","last_became_active":"2024-03-19T01:23:27.538339+0000","last_became_peered":"2024-03-19T01:23:27.538339+0000","last_unstale":"2024-03-19T01:23:28.498214+0000","last_undegraded":"2024-03-19T01:23:28.498214+0000","last_fullsized":"2024-03-19T01:23:28.498214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.500126+0000","last_change":"2024-03-19T01:23:27.562551+0000","last_active":"2024-03-19T01:23:28.500126+0000","last_peered":"2024-03-19T01:23:28.500126+0000","last_clean":"2024-03-19T01:23:28.500126+0000","last_became_active":"2024-03-19T01:23:27.561996+0000","last_became_peered":"2024-03-19T01:23:27.561996+0000","last_unstale":"2024-03-19T01:23:28.500126+0000","last_undegraded":"2024-03-19T01:23:28.500126+0000","last_fullsized":"2024-03-19T01:23:28.500126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:27.331497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572895+0000","last_change":"2024-03-19T01:23:27.541038+0000","last_active":"2024-03-19T01:23:28.572895+0000","last_peered":"2024-03-19T01:23:28.572895+0000","last_clean":"2024-03-19T01:23:28.572895+0000","last_became_active":"2024-03-19T01:23:27.540751+0000","last_became_peered":"2024-03-19T01:23:27.540751+0000","last_unstale":"2024-03-19T01:23:28.572895+0000","last_undegraded":"2024-03-19T01:23:28.572895+0000","last_fullsized":"2024-03-19T01:23:28.572895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.559025+0000","last_change":"2024-03-19T01:23:27.552053+0000","last_active":"2024-03-19T01:23:27.559025+0000","last_peered":"2024-03-19T01:23:27.559025+0000","last_clean":"2024-03-19T01:23:27.559025+0000","last_became_active":"2024-03-19T01:23:27.551405+0000","last_became_peered":"2024-03-19T01:23:27.551405+0000","last_unstale":"2024-03-19T01:23:27.559025+0000","last_undegraded":"2024-03-19T01:23:27.559025+0000","last_fullsized":"2024-03-19T01:23:27.559025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.553016+0000","last_change":"2024-03-19T01:23:27.548149+0000","last_active":"2024-03-19T01:23:27.553016+0000","last_peered":"2024-03-19T01:23:27.553016+0000","last_clean":"2024-03-19T01:23:27.553016+0000","last_became_active":"2024-03-19T01:23:27.542105+0000","last_became_peered":"2024-03-19T01:23:27.542105+0000","last_unstale":"2024-03-19T01:23:27.553016+0000","last_undegraded":"2024-03-19T01:23:27.553016+0000","last_fullsized":"2024-03-19T01:23:27.553016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:00.911501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.523235+0000","last_change":"2024-03-19T01:23:27.521814+0000","last_active":"2024-03-19T01:23:27.523235+0000","last_peered":"2024-03-19T01:23:27.523235+0000","last_clean":"2024-03-19T01:23:27.523235+0000","last_became_active":"2024-03-19T01:23:27.519639+0000","last_became_peered":"2024-03-19T01:23:27.519639+0000","last_unstale":"2024-03-19T01:23:27.523235+0000","last_undegraded":"2024-03-19T01:23:27.523235+0000","last_fullsized":"2024-03-19T01:23:27.523235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:08.835543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499387+0000","last_change":"2024-03-19T01:23:27.558248+0000","last_active":"2024-03-19T01:23:28.499387+0000","last_peered":"2024-03-19T01:23:28.499387+0000","last_clean":"2024-03-19T01:23:28.499387+0000","last_became_active":"2024-03-19T01:23:27.557663+0000","last_became_peered":"2024-03-19T01:23:27.557663+0000","last_unstale":"2024-03-19T01:23:28.499387+0000","last_undegraded":"2024-03-19T01:23:28.499387+0000","last_fullsized":"2024-03-19T01:23:28.499387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.568928+0000","last_change":"2024-03-19T01:23:27.532119+0000","last_active":"2024-03-19T01:23:28.568928+0000","last_peered":"2024-03-19T01:23:28.568928+0000","last_clean":"2024-03-19T01:23:28.568928+0000","last_became_active":"2024-03-19T01:23:27.531846+0000","last_became_peered":"2024-03-19T01:23:27.531846+0000","last_unstale":"2024-03-19T01:23:28.568928+0000","last_undegraded":"2024-03-19T01:23:28.568928+0000","last_fullsized":"2024-03-19T01:23:28.568928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498373+0000","last_change":"2024-03-19T01:23:27.546292+0000","last_active":"2024-03-19T01:23:28.498373+0000","last_peered":"2024-03-19T01:23:28.498373+0000","last_clean":"2024-03-19T01:23:28.498373+0000","last_became_active":"2024-03-19T01:23:27.537369+0000","last_became_peered":"2024-03-19T01:23:27.537369+0000","last_unstale":"2024-03-19T01:23:28.498373+0000","last_undegraded":"2024-03-19T01:23:28.498373+0000","last_fullsized":"2024-03-19T01:23:28.498373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569601+0000","last_change":"2024-03-19T01:23:27.522448+0000","last_active":"2024-03-19T01:23:28.569601+0000","last_peered":"2024-03-19T01:23:28.569601+0000","last_clean":"2024-03-19T01:23:28.569601+0000","last_became_active":"2024-03-19T01:23:27.522201+0000","last_became_peered":"2024-03-19T01:23:27.522201+0000","last_unstale":"2024-03-19T01:23:28.569601+0000","last_undegraded":"2024-03-19T01:23:28.569601+0000","last_fullsized":"2024-03-19T01:23:28.569601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.553016+0000","last_change":"2024-03-19T01:23:27.551178+0000","last_active":"2024-03-19T01:23:27.553016+0000","last_peered":"2024-03-19T01:23:27.553016+0000","last_clean":"2024-03-19T01:23:27.553016+0000","last_became_active":"2024-03-19T01:23:27.550571+0000","last_became_peered":"2024-03-19T01:23:27.550571+0000","last_unstale":"2024-03-19T01:23:27.553016+0000","last_undegraded":"2024-03-19T01:23:27.553016+0000","last_fullsized":"2024-03-19T01:23:27.553016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:52.938759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497894+0000","last_change":"2024-03-19T01:23:27.509194+0000","last_active":"2024-03-19T01:23:28.497894+0000","last_peered":"2024-03-19T01:23:28.497894+0000","last_clean":"2024-03-19T01:23:28.497894+0000","last_became_active":"2024-03-19T01:23:27.508796+0000","last_became_peered":"2024-03-19T01:23:27.508796+0000","last_unstale":"2024-03-19T01:23:28.497894+0000","last_undegraded":"2024-03-19T01:23:28.497894+0000","last_fullsized":"2024-03-19T01:23:28.497894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569258+0000","last_change":"2024-03-19T01:23:27.522380+0000","last_active":"2024-03-19T01:23:28.569258+0000","last_peered":"2024-03-19T01:23:28.569258+0000","last_clean":"2024-03-19T01:23:28.569258+0000","last_became_active":"2024-03-19T01:23:27.521968+0000","last_became_peered":"2024-03-19T01:23:27.521968+0000","last_unstale":"2024-03-19T01:23:28.569258+0000","last_undegraded":"2024-03-19T01:23:28.569258+0000","last_fullsized":"2024-03-19T01:23:28.569258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.519923+0000","last_change":"2024-03-19T01:23:27.516056+0000","last_active":"2024-03-19T01:23:27.519923+0000","last_peered":"2024-03-19T01:23:27.519923+0000","last_clean":"2024-03-19T01:23:27.519923+0000","last_became_active":"2024-03-19T01:23:27.515651+0000","last_became_peered":"2024-03-19T01:23:27.515651+0000","last_unstale":"2024-03-19T01:23:27.519923+0000","last_undegraded":"2024-03-19T01:23:27.519923+0000","last_fullsized":"2024-03-19T01:23:27.519923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:38.911150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499052+0000","last_change":"2024-03-19T01:23:27.547751+0000","last_active":"2024-03-19T01:23:28.499052+0000","last_peered":"2024-03-19T01:23:28.499052+0000","last_clean":"2024-03-19T01:23:28.499052+0000","last_became_active":"2024-03-19T01:23:27.530027+0000","last_became_peered":"2024-03-19T01:23:27.530027+0000","last_unstale":"2024-03-19T01:23:28.499052+0000","last_undegraded":"2024-03-19T01:23:28.499052+0000","last_fullsized":"2024-03-19T01:23:28.499052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572767+0000","last_change":"2024-03-19T01:23:27.538411+0000","last_active":"2024-03-19T01:23:28.572767+0000","last_peered":"2024-03-19T01:23:28.572767+0000","last_clean":"2024-03-19T01:23:28.572767+0000","last_became_active":"2024-03-19T01:23:27.537893+0000","last_became_peered":"2024-03-19T01:23:27.537893+0000","last_unstale":"2024-03-19T01:23:28.572767+0000","last_undegraded":"2024-03-19T01:23:28.572767+0000","last_fullsized":"2024-03-19T01:23:28.572767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.548480+0000","last_change":"2024-03-19T01:23:27.539735+0000","last_active":"2024-03-19T01:23:27.548480+0000","last_peered":"2024-03-19T01:23:27.548480+0000","last_clean":"2024-03-19T01:23:27.548480+0000","last_became_active":"2024-03-19T01:23:27.539022+0000","last_became_peered":"2024-03-19T01:23:27.539022+0000","last_unstale":"2024-03-19T01:23:27.548480+0000","last_undegraded":"2024-03-19T01:23:27.548480+0000","last_fullsized":"2024-03-19T01:23:27.548480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:08.835543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830822+0000","last_change":"2024-03-19T01:23:27.503767+0000","last_active":"2024-03-19T01:23:28.830822+0000","last_peered":"2024-03-19T01:23:28.830822+0000","last_clean":"2024-03-19T01:23:28.830822+0000","last_became_active":"2024-03-19T01:23:27.503030+0000","last_became_peered":"2024-03-19T01:23:27.503030+0000","last_unstale":"2024-03-19T01:23:28.830822+0000","last_undegraded":"2024-03-19T01:23:28.830822+0000","last_fullsized":"2024-03-19T01:23:28.830822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:08.835543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569515+0000","last_change":"2024-03-19T01:23:26.532832+0000","last_active":"2024-03-19T01:23:28.569515+0000","last_peered":"2024-03-19T01:23:28.569515+0000","last_clean":"2024-03-19T01:23:28.569515+0000","last_became_active":"2024-03-19T01:23:26.532516+0000","last_became_peered":"2024-03-19T01:23:26.532516+0000","last_unstale":"2024-03-19T01:23:28.569515+0000","last_undegraded":"2024-03-19T01:23:28.569515+0000","last_fullsized":"2024-03-19T01:23:28.569515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.523812+0000","last_change":"2024-03-19T01:23:27.522605+0000","last_active":"2024-03-19T01:23:27.523812+0000","last_peered":"2024-03-19T01:23:27.523812+0000","last_clean":"2024-03-19T01:23:27.523812+0000","last_became_active":"2024-03-19T01:23:27.522344+0000","last_became_peered":"2024-03-19T01:23:27.522344+0000","last_unstale":"2024-03-19T01:23:27.523812+0000","last_undegraded":"2024-03-19T01:23:27.523812+0000","last_fullsized":"2024-03-19T01:23:27.523812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.547716+0000","last_change":"2024-03-19T01:23:26.535891+0000","last_active":"2024-03-19T01:23:27.547716+0000","last_peered":"2024-03-19T01:23:27.547716+0000","last_clean":"2024-03-19T01:23:27.547716+0000","last_became_active":"2024-03-19T01:23:26.535218+0000","last_became_peered":"2024-03-19T01:23:26.535218+0000","last_unstale":"2024-03-19T01:23:27.547716+0000","last_undegraded":"2024-03-19T01:23:27.547716+0000","last_fullsized":"2024-03-19T01:23:27.547716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.568845+0000","last_change":"2024-03-19T01:23:26.536647+0000","last_active":"2024-03-19T01:23:28.568845+0000","last_peered":"2024-03-19T01:23:28.568845+0000","last_clean":"2024-03-19T01:23:28.568845+0000","last_became_active":"2024-03-19T01:23:26.535919+0000","last_became_peered":"2024-03-19T01:23:26.535919+0000","last_unstale":"2024-03-19T01:23:28.568845+0000","last_undegraded":"2024-03-19T01:23:28.568845+0000","last_fullsized":"2024-03-19T01:23:28.568845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569044+0000","last_change":"2024-03-19T01:23:27.517889+0000","last_active":"2024-03-19T01:23:28.569044+0000","last_peered":"2024-03-19T01:23:28.569044+0000","last_clean":"2024-03-19T01:23:28.569044+0000","last_became_active":"2024-03-19T01:23:27.515221+0000","last_became_peered":"2024-03-19T01:23:27.515221+0000","last_unstale":"2024-03-19T01:23:28.569044+0000","last_undegraded":"2024-03-19T01:23:28.569044+0000","last_fullsized":"2024-03-19T01:23:28.569044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499348+0000","last_change":"2024-03-19T01:23:26.518817+0000","last_active":"2024-03-19T01:23:28.499348+0000","last_peered":"2024-03-19T01:23:28.499348+0000","last_clean":"2024-03-19T01:23:28.499348+0000","last_became_active":"2024-03-19T01:23:26.518008+0000","last_became_peered":"2024-03-19T01:23:26.518008+0000","last_unstale":"2024-03-19T01:23:28.499348+0000","last_undegraded":"2024-03-19T01:23:28.499348+0000","last_fullsized":"2024-03-19T01:23:28.499348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498152+0000","last_change":"2024-03-19T01:23:27.514550+0000","last_active":"2024-03-19T01:23:28.498152+0000","last_peered":"2024-03-19T01:23:28.498152+0000","last_clean":"2024-03-19T01:23:28.498152+0000","last_became_active":"2024-03-19T01:23:27.512624+0000","last_became_peered":"2024-03-19T01:23:27.512624+0000","last_unstale":"2024-03-19T01:23:28.498152+0000","last_undegraded":"2024-03-19T01:23:28.498152+0000","last_fullsized":"2024-03-19T01:23:28.498152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.503744+0000","last_change":"2024-03-19T01:23:26.536461+0000","last_active":"2024-03-19T01:23:27.503744+0000","last_peered":"2024-03-19T01:23:27.503744+0000","last_clean":"2024-03-19T01:23:27.503744+0000","last_became_active":"2024-03-19T01:23:26.535707+0000","last_became_peered":"2024-03-19T01:23:26.535707+0000","last_unstale":"2024-03-19T01:23:27.503744+0000","last_undegraded":"2024-03-19T01:23:27.503744+0000","last_fullsized":"2024-03-19T01:23:27.503744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499599+0000","last_change":"2024-03-19T01:23:27.561680+0000","last_active":"2024-03-19T01:23:28.499599+0000","last_peered":"2024-03-19T01:23:28.499599+0000","last_clean":"2024-03-19T01:23:28.499599+0000","last_became_active":"2024-03-19T01:23:27.559667+0000","last_became_peered":"2024-03-19T01:23:27.559667+0000","last_unstale":"2024-03-19T01:23:28.499599+0000","last_undegraded":"2024-03-19T01:23:28.499599+0000","last_fullsized":"2024-03-19T01:23:28.499599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:28.568774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499535+0000","last_change":"2024-03-19T01:23:26.541115+0000","last_active":"2024-03-19T01:23:28.499535+0000","last_peered":"2024-03-19T01:23:28.499535+0000","last_clean":"2024-03-19T01:23:28.499535+0000","last_became_active":"2024-03-19T01:23:26.540408+0000","last_became_peered":"2024-03-19T01:23:26.540408+0000","last_unstale":"2024-03-19T01:23:28.499535+0000","last_undegraded":"2024-03-19T01:23:28.499535+0000","last_fullsized":"2024-03-19T01:23:28.499535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.558153+0000","last_change":"2024-03-19T01:23:27.557631+0000","last_active":"2024-03-19T01:23:27.558153+0000","last_peered":"2024-03-19T01:23:27.558153+0000","last_clean":"2024-03-19T01:23:27.558153+0000","last_became_active":"2024-03-19T01:23:27.557324+0000","last_became_peered":"2024-03-19T01:23:27.557324+0000","last_unstale":"2024-03-19T01:23:27.558153+0000","last_undegraded":"2024-03-19T01:23:27.558153+0000","last_fullsized":"2024-03-19T01:23:27.558153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:23.802985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498202+0000","last_change":"2024-03-19T01:23:26.517007+0000","last_active":"2024-03-19T01:23:28.498202+0000","last_peered":"2024-03-19T01:23:28.498202+0000","last_clean":"2024-03-19T01:23:28.498202+0000","last_became_active":"2024-03-19T01:23:26.516710+0000","last_became_peered":"2024-03-19T01:23:26.516710+0000","last_unstale":"2024-03-19T01:23:28.498202+0000","last_undegraded":"2024-03-19T01:23:28.498202+0000","last_fullsized":"2024-03-19T01:23:28.498202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:36.870147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831243+0000","last_change":"2024-03-19T01:23:27.515805+0000","last_active":"2024-03-19T01:23:28.831243+0000","last_peered":"2024-03-19T01:23:28.831243+0000","last_clean":"2024-03-19T01:23:28.831243+0000","last_became_active":"2024-03-19T01:23:27.515099+0000","last_became_peered":"2024-03-19T01:23:27.515099+0000","last_unstale":"2024-03-19T01:23:28.831243+0000","last_undegraded":"2024-03-19T01:23:28.831243+0000","last_fullsized":"2024-03-19T01:23:28.831243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:00.911501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.557475+0000","last_change":"2024-03-19T01:23:26.535383+0000","last_active":"2024-03-19T01:23:27.557475+0000","last_peered":"2024-03-19T01:23:27.557475+0000","last_clean":"2024-03-19T01:23:27.557475+0000","last_became_active":"2024-03-19T01:23:26.534052+0000","last_became_peered":"2024-03-19T01:23:26.534052+0000","last_unstale":"2024-03-19T01:23:27.557475+0000","last_undegraded":"2024-03-19T01:23:27.557475+0000","last_fullsized":"2024-03-19T01:23:27.557475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.558988+0000","last_change":"2024-03-19T01:23:27.550833+0000","last_active":"2024-03-19T01:23:27.558988+0000","last_peered":"2024-03-19T01:23:27.558988+0000","last_clean":"2024-03-19T01:23:27.558988+0000","last_became_active":"2024-03-19T01:23:27.544992+0000","last_became_peered":"2024-03-19T01:23:27.544992+0000","last_unstale":"2024-03-19T01:23:27.558988+0000","last_undegraded":"2024-03-19T01:23:27.558988+0000","last_fullsized":"2024-03-19T01:23:27.558988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.526141+0000","last_change":"2024-03-19T01:23:26.506460+0000","last_active":"2024-03-19T01:23:27.526141+0000","last_peered":"2024-03-19T01:23:27.526141+0000","last_clean":"2024-03-19T01:23:27.526141+0000","last_became_active":"2024-03-19T01:23:26.506155+0000","last_became_peered":"2024-03-19T01:23:26.506155+0000","last_unstale":"2024-03-19T01:23:27.526141+0000","last_undegraded":"2024-03-19T01:23:27.526141+0000","last_fullsized":"2024-03-19T01:23:27.526141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:10.187739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.515234+0000","last_change":"2024-03-19T01:23:27.514233+0000","last_active":"2024-03-19T01:23:27.515234+0000","last_peered":"2024-03-19T01:23:27.515234+0000","last_clean":"2024-03-19T01:23:27.515234+0000","last_became_active":"2024-03-19T01:23:27.513950+0000","last_became_peered":"2024-03-19T01:23:27.513950+0000","last_unstale":"2024-03-19T01:23:27.515234+0000","last_undegraded":"2024-03-19T01:23:27.515234+0000","last_fullsized":"2024-03-19T01:23:27.515234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:51.109900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499325+0000","last_change":"2024-03-19T01:23:26.516202+0000","last_active":"2024-03-19T01:23:28.499325+0000","last_peered":"2024-03-19T01:23:28.499325+0000","last_clean":"2024-03-19T01:23:28.499325+0000","last_became_active":"2024-03-19T01:23:26.515081+0000","last_became_peered":"2024-03-19T01:23:26.515081+0000","last_unstale":"2024-03-19T01:23:28.499325+0000","last_undegraded":"2024-03-19T01:23:28.499325+0000","last_fullsized":"2024-03-19T01:23:28.499325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.555944+0000","last_change":"2024-03-19T01:23:27.534410+0000","last_active":"2024-03-19T01:23:27.555944+0000","last_peered":"2024-03-19T01:23:27.555944+0000","last_clean":"2024-03-19T01:23:27.555944+0000","last_became_active":"2024-03-19T01:23:27.533583+0000","last_became_peered":"2024-03-19T01:23:27.533583+0000","last_unstale":"2024-03-19T01:23:27.555944+0000","last_undegraded":"2024-03-19T01:23:27.555944+0000","last_fullsized":"2024-03-19T01:23:27.555944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:23.802985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.505196+0000","last_change":"2024-03-19T01:23:26.530022+0000","last_active":"2024-03-19T01:23:27.505196+0000","last_peered":"2024-03-19T01:23:27.505196+0000","last_clean":"2024-03-19T01:23:27.505196+0000","last_became_active":"2024-03-19T01:23:26.525529+0000","last_became_peered":"2024-03-19T01:23:26.525529+0000","last_unstale":"2024-03-19T01:23:27.505196+0000","last_undegraded":"2024-03-19T01:23:27.505196+0000","last_fullsized":"2024-03-19T01:23:27.505196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.551502+0000","last_change":"2024-03-19T01:23:27.542939+0000","last_active":"2024-03-19T01:23:27.551502+0000","last_peered":"2024-03-19T01:23:27.551502+0000","last_clean":"2024-03-19T01:23:27.551502+0000","last_became_active":"2024-03-19T01:23:27.532364+0000","last_became_peered":"2024-03-19T01:23:27.532364+0000","last_unstale":"2024-03-19T01:23:27.551502+0000","last_undegraded":"2024-03-19T01:23:27.551502+0000","last_fullsized":"2024-03-19T01:23:27.551502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572800+0000","last_change":"2024-03-19T01:23:26.513589+0000","last_active":"2024-03-19T01:23:28.572800+0000","last_peered":"2024-03-19T01:23:28.572800+0000","last_clean":"2024-03-19T01:23:28.572800+0000","last_became_active":"2024-03-19T01:23:26.513223+0000","last_became_peered":"2024-03-19T01:23:26.513223+0000","last_unstale":"2024-03-19T01:23:28.572800+0000","last_undegraded":"2024-03-19T01:23:28.572800+0000","last_fullsized":"2024-03-19T01:23:28.572800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.558933+0000","last_change":"2024-03-19T01:23:27.552435+0000","last_active":"2024-03-19T01:23:27.558933+0000","last_peered":"2024-03-19T01:23:27.558933+0000","last_clean":"2024-03-19T01:23:27.558933+0000","last_became_active":"2024-03-19T01:23:27.545233+0000","last_became_peered":"2024-03-19T01:23:27.545233+0000","last_unstale":"2024-03-19T01:23:27.558933+0000","last_undegraded":"2024-03-19T01:23:27.558933+0000","last_fullsized":"2024-03-19T01:23:27.558933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831253+0000","last_change":"2024-03-19T01:23:26.515010+0000","last_active":"2024-03-19T01:23:28.831253+0000","last_peered":"2024-03-19T01:23:28.831253+0000","last_clean":"2024-03-19T01:23:28.831253+0000","last_became_active":"2024-03-19T01:23:26.513690+0000","last_became_peered":"2024-03-19T01:23:26.513690+0000","last_unstale":"2024-03-19T01:23:28.831253+0000","last_undegraded":"2024-03-19T01:23:28.831253+0000","last_fullsized":"2024-03-19T01:23:28.831253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:41:56.780781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556636+0000","last_change":"2024-03-19T01:23:27.535739+0000","last_active":"2024-03-19T01:23:27.556636+0000","last_peered":"2024-03-19T01:23:27.556636+0000","last_clean":"2024-03-19T01:23:27.556636+0000","last_became_active":"2024-03-19T01:23:27.534174+0000","last_became_peered":"2024-03-19T01:23:27.534174+0000","last_unstale":"2024-03-19T01:23:27.556636+0000","last_undegraded":"2024-03-19T01:23:27.556636+0000","last_fullsized":"2024-03-19T01:23:27.556636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:52.938759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572994+0000","last_change":"2024-03-19T01:23:26.512861+0000","last_active":"2024-03-19T01:23:28.572994+0000","last_peered":"2024-03-19T01:23:28.572994+0000","last_clean":"2024-03-19T01:23:28.572994+0000","last_became_active":"2024-03-19T01:23:26.512475+0000","last_became_peered":"2024-03-19T01:23:26.512475+0000","last_unstale":"2024-03-19T01:23:28.572994+0000","last_undegraded":"2024-03-19T01:23:28.572994+0000","last_fullsized":"2024-03-19T01:23:28.572994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.560392+0000","last_change":"2024-03-19T01:23:27.553761+0000","last_active":"2024-03-19T01:23:27.560392+0000","last_peered":"2024-03-19T01:23:27.560392+0000","last_clean":"2024-03-19T01:23:27.560392+0000","last_became_active":"2024-03-19T01:23:27.551510+0000","last_became_peered":"2024-03-19T01:23:27.551510+0000","last_unstale":"2024-03-19T01:23:27.560392+0000","last_undegraded":"2024-03-19T01:23:27.560392+0000","last_fullsized":"2024-03-19T01:23:27.560392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498228+0000","last_change":"2024-03-19T01:23:26.520079+0000","last_active":"2024-03-19T01:23:28.498228+0000","last_peered":"2024-03-19T01:23:28.498228+0000","last_clean":"2024-03-19T01:23:28.498228+0000","last_became_active":"2024-03-19T01:23:26.519678+0000","last_became_peered":"2024-03-19T01:23:26.519678+0000","last_unstale":"2024-03-19T01:23:28.498228+0000","last_undegraded":"2024-03-19T01:23:28.498228+0000","last_fullsized":"2024-03-19T01:23:28.498228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:42.729108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499902+0000","last_change":"2024-03-19T01:23:27.535733+0000","last_active":"2024-03-19T01:23:28.499902+0000","last_peered":"2024-03-19T01:23:28.499902+0000","last_clean":"2024-03-19T01:23:28.499902+0000","last_became_active":"2024-03-19T01:23:27.529249+0000","last_became_peered":"2024-03-19T01:23:27.529249+0000","last_unstale":"2024-03-19T01:23:28.499902+0000","last_undegraded":"2024-03-19T01:23:28.499902+0000","last_fullsized":"2024-03-19T01:23:28.499902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:33.934449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572947+0000","last_change":"2024-03-19T01:23:26.518355+0000","last_active":"2024-03-19T01:23:28.572947+0000","last_peered":"2024-03-19T01:23:28.572947+0000","last_clean":"2024-03-19T01:23:28.572947+0000","last_became_active":"2024-03-19T01:23:26.517357+0000","last_became_peered":"2024-03-19T01:23:26.517357+0000","last_unstale":"2024-03-19T01:23:28.572947+0000","last_undegraded":"2024-03-19T01:23:28.572947+0000","last_fullsized":"2024-03-19T01:23:28.572947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.565860+0000","last_change":"2024-03-19T01:23:27.564792+0000","last_active":"2024-03-19T01:23:27.565860+0000","last_peered":"2024-03-19T01:23:27.565860+0000","last_clean":"2024-03-19T01:23:27.565860+0000","last_became_active":"2024-03-19T01:23:27.564199+0000","last_became_peered":"2024-03-19T01:23:27.564199+0000","last_unstale":"2024-03-19T01:23:27.565860+0000","last_undegraded":"2024-03-19T01:23:27.565860+0000","last_fullsized":"2024-03-19T01:23:27.565860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.507342+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:23:27.507342+0000","last_peered":"2024-03-19T01:23:27.507342+0000","last_clean":"2024-03-19T01:23:27.507342+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:23:27.507342+0000","last_undegraded":"2024-03-19T01:23:27.507342+0000","last_fullsized":"2024-03-19T01:23:27.507342+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498418+0000","last_change":"2024-03-19T01:23:26.516922+0000","last_active":"2024-03-19T01:23:28.498418+0000","last_peered":"2024-03-19T01:23:28.498418+0000","last_clean":"2024-03-19T01:23:28.498418+0000","last_became_active":"2024-03-19T01:23:26.516206+0000","last_became_peered":"2024-03-19T01:23:26.516206+0000","last_unstale":"2024-03-19T01:23:28.498418+0000","last_undegraded":"2024-03-19T01:23:28.498418+0000","last_fullsized":"2024-03-19T01:23:28.498418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569215+0000","last_change":"2024-03-19T01:23:27.557742+0000","last_active":"2024-03-19T01:23:28.569215+0000","last_peered":"2024-03-19T01:23:28.569215+0000","last_clean":"2024-03-19T01:23:28.569215+0000","last_became_active":"2024-03-19T01:23:27.557487+0000","last_became_peered":"2024-03-19T01:23:27.557487+0000","last_unstale":"2024-03-19T01:23:28.569215+0000","last_undegraded":"2024-03-19T01:23:28.569215+0000","last_fullsized":"2024-03-19T01:23:28.569215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.507514+0000","last_change":"2024-03-19T01:23:26.539730+0000","last_active":"2024-03-19T01:23:27.507514+0000","last_peered":"2024-03-19T01:23:27.507514+0000","last_clean":"2024-03-19T01:23:27.507514+0000","last_became_active":"2024-03-19T01:23:26.538551+0000","last_became_peered":"2024-03-19T01:23:26.538551+0000","last_unstale":"2024-03-19T01:23:27.507514+0000","last_undegraded":"2024-03-19T01:23:27.507514+0000","last_fullsized":"2024-03-19T01:23:27.507514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.524654+0000","last_change":"2024-03-19T01:23:27.524267+0000","last_active":"2024-03-19T01:23:27.524654+0000","last_peered":"2024-03-19T01:23:27.524654+0000","last_clean":"2024-03-19T01:23:27.524654+0000","last_became_active":"2024-03-19T01:23:27.523862+0000","last_became_peered":"2024-03-19T01:23:27.523862+0000","last_unstale":"2024-03-19T01:23:27.524654+0000","last_undegraded":"2024-03-19T01:23:27.524654+0000","last_fullsized":"2024-03-19T01:23:27.524654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572805+0000","last_change":"2024-03-19T01:23:26.516027+0000","last_active":"2024-03-19T01:23:28.572805+0000","last_peered":"2024-03-19T01:23:28.572805+0000","last_clean":"2024-03-19T01:23:28.572805+0000","last_became_active":"2024-03-19T01:23:26.515582+0000","last_became_peered":"2024-03-19T01:23:26.515582+0000","last_unstale":"2024-03-19T01:23:28.572805+0000","last_undegraded":"2024-03-19T01:23:28.572805+0000","last_fullsized":"2024-03-19T01:23:28.572805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498092+0000","last_change":"2024-03-19T01:23:27.515653+0000","last_active":"2024-03-19T01:23:28.498092+0000","last_peered":"2024-03-19T01:23:28.498092+0000","last_clean":"2024-03-19T01:23:28.498092+0000","last_became_active":"2024-03-19T01:23:27.511685+0000","last_became_peered":"2024-03-19T01:23:27.511685+0000","last_unstale":"2024-03-19T01:23:28.498092+0000","last_undegraded":"2024-03-19T01:23:28.498092+0000","last_fullsized":"2024-03-19T01:23:28.498092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499532+0000","last_change":"2024-03-19T01:23:26.511797+0000","last_active":"2024-03-19T01:23:28.499532+0000","last_peered":"2024-03-19T01:23:28.499532+0000","last_clean":"2024-03-19T01:23:28.499532+0000","last_became_active":"2024-03-19T01:23:26.511112+0000","last_became_peered":"2024-03-19T01:23:26.511112+0000","last_unstale":"2024-03-19T01:23:28.499532+0000","last_undegraded":"2024-03-19T01:23:28.499532+0000","last_fullsized":"2024-03-19T01:23:28.499532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830892+0000","last_change":"2024-03-19T01:23:27.523591+0000","last_active":"2024-03-19T01:23:28.830892+0000","last_peered":"2024-03-19T01:23:28.830892+0000","last_clean":"2024-03-19T01:23:28.830892+0000","last_became_active":"2024-03-19T01:23:27.522418+0000","last_became_peered":"2024-03-19T01:23:27.522418+0000","last_unstale":"2024-03-19T01:23:28.830892+0000","last_undegraded":"2024-03-19T01:23:28.830892+0000","last_fullsized":"2024-03-19T01:23:28.830892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498246+0000","last_change":"2024-03-19T01:23:26.519145+0000","last_active":"2024-03-19T01:23:28.498246+0000","last_peered":"2024-03-19T01:23:28.498246+0000","last_clean":"2024-03-19T01:23:28.498246+0000","last_became_active":"2024-03-19T01:23:26.518686+0000","last_became_peered":"2024-03-19T01:23:26.518686+0000","last_unstale":"2024-03-19T01:23:28.498246+0000","last_undegraded":"2024-03-19T01:23:28.498246+0000","last_fullsized":"2024-03-19T01:23:28.498246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499719+0000","last_change":"2024-03-19T01:23:27.549327+0000","last_active":"2024-03-19T01:23:28.499719+0000","last_peered":"2024-03-19T01:23:28.499719+0000","last_clean":"2024-03-19T01:23:28.499719+0000","last_became_active":"2024-03-19T01:23:27.548727+0000","last_became_peered":"2024-03-19T01:23:27.548727+0000","last_unstale":"2024-03-19T01:23:28.499719+0000","last_undegraded":"2024-03-19T01:23:28.499719+0000","last_fullsized":"2024-03-19T01:23:28.499719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:56.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.524302+0000","last_change":"2024-03-19T01:23:26.539153+0000","last_active":"2024-03-19T01:23:27.524302+0000","last_peered":"2024-03-19T01:23:27.524302+0000","last_clean":"2024-03-19T01:23:27.524302+0000","last_became_active":"2024-03-19T01:23:26.537687+0000","last_became_peered":"2024-03-19T01:23:26.537687+0000","last_unstale":"2024-03-19T01:23:27.524302+0000","last_undegraded":"2024-03-19T01:23:27.524302+0000","last_fullsized":"2024-03-19T01:23:27.524302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:59.899930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497941+0000","last_change":"2024-03-19T01:23:27.508732+0000","last_active":"2024-03-19T01:23:28.497941+0000","last_peered":"2024-03-19T01:23:28.497941+0000","last_clean":"2024-03-19T01:23:28.497941+0000","last_became_active":"2024-03-19T01:23:27.508329+0000","last_became_peered":"2024-03-19T01:23:27.508329+0000","last_unstale":"2024-03-19T01:23:28.497941+0000","last_undegraded":"2024-03-19T01:23:28.497941+0000","last_fullsized":"2024-03-19T01:23:28.497941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498756+0000","last_change":"2024-03-19T01:23:26.536445+0000","last_active":"2024-03-19T01:23:28.498756+0000","last_peered":"2024-03-19T01:23:28.498756+0000","last_clean":"2024-03-19T01:23:28.498756+0000","last_became_active":"2024-03-19T01:23:26.535968+0000","last_became_peered":"2024-03-19T01:23:26.535968+0000","last_unstale":"2024-03-19T01:23:28.498756+0000","last_undegraded":"2024-03-19T01:23:28.498756+0000","last_fullsized":"2024-03-19T01:23:28.498756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569863+0000","last_change":"2024-03-19T01:23:27.553070+0000","last_active":"2024-03-19T01:23:28.569863+0000","last_peered":"2024-03-19T01:23:28.569863+0000","last_clean":"2024-03-19T01:23:28.569863+0000","last_became_active":"2024-03-19T01:23:27.552794+0000","last_became_peered":"2024-03-19T01:23:27.552794+0000","last_unstale":"2024-03-19T01:23:28.569863+0000","last_undegraded":"2024-03-19T01:23:28.569863+0000","last_fullsized":"2024-03-19T01:23:28.569863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.554475+0000","last_change":"2024-03-19T01:23:26.537527+0000","last_active":"2024-03-19T01:23:27.554475+0000","last_peered":"2024-03-19T01:23:27.554475+0000","last_clean":"2024-03-19T01:23:27.554475+0000","last_became_active":"2024-03-19T01:23:26.536523+0000","last_became_peered":"2024-03-19T01:23:26.536523+0000","last_unstale":"2024-03-19T01:23:27.554475+0000","last_undegraded":"2024-03-19T01:23:27.554475+0000","last_fullsized":"2024-03-19T01:23:27.554475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.514918+0000","last_change":"2024-03-19T01:23:27.511086+0000","last_active":"2024-03-19T01:23:27.514918+0000","last_peered":"2024-03-19T01:23:27.514918+0000","last_clean":"2024-03-19T01:23:27.514918+0000","last_became_active":"2024-03-19T01:23:27.510714+0000","last_became_peered":"2024-03-19T01:23:27.510714+0000","last_unstale":"2024-03-19T01:23:27.514918+0000","last_undegraded":"2024-03-19T01:23:27.514918+0000","last_fullsized":"2024-03-19T01:23:27.514918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:15.544112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.503246+0000","last_change":"2024-03-19T01:23:26.539154+0000","last_active":"2024-03-19T01:23:27.503246+0000","last_peered":"2024-03-19T01:23:27.503246+0000","last_clean":"2024-03-19T01:23:27.503246+0000","last_became_active":"2024-03-19T01:23:26.537837+0000","last_became_peered":"2024-03-19T01:23:26.537837+0000","last_unstale":"2024-03-19T01:23:27.503246+0000","last_undegraded":"2024-03-19T01:23:27.503246+0000","last_fullsized":"2024-03-19T01:23:27.503246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830757+0000","last_change":"2024-03-19T01:23:27.512029+0000","last_active":"2024-03-19T01:23:28.830757+0000","last_peered":"2024-03-19T01:23:28.830757+0000","last_clean":"2024-03-19T01:23:28.830757+0000","last_became_active":"2024-03-19T01:23:27.511155+0000","last_became_peered":"2024-03-19T01:23:27.511155+0000","last_unstale":"2024-03-19T01:23:28.830757+0000","last_undegraded":"2024-03-19T01:23:28.830757+0000","last_fullsized":"2024-03-19T01:23:28.830757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:38.911150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497954+0000","last_change":"2024-03-19T01:23:26.528143+0000","last_active":"2024-03-19T01:23:28.497954+0000","last_peered":"2024-03-19T01:23:28.497954+0000","last_clean":"2024-03-19T01:23:28.497954+0000","last_became_active":"2024-03-19T01:23:26.527487+0000","last_became_peered":"2024-03-19T01:23:26.527487+0000","last_unstale":"2024-03-19T01:23:28.497954+0000","last_undegraded":"2024-03-19T01:23:28.497954+0000","last_fullsized":"2024-03-19T01:23:28.497954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:07.823972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569510+0000","last_change":"2024-03-19T01:23:27.518443+0000","last_active":"2024-03-19T01:23:28.569510+0000","last_peered":"2024-03-19T01:23:28.569510+0000","last_clean":"2024-03-19T01:23:28.569510+0000","last_became_active":"2024-03-19T01:23:27.518170+0000","last_became_peered":"2024-03-19T01:23:27.518170+0000","last_unstale":"2024-03-19T01:23:28.569510+0000","last_undegraded":"2024-03-19T01:23:28.569510+0000","last_fullsized":"2024-03-19T01:23:28.569510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498375+0000","last_change":"2024-03-19T01:23:26.511877+0000","last_active":"2024-03-19T01:23:28.498375+0000","last_peered":"2024-03-19T01:23:28.498375+0000","last_clean":"2024-03-19T01:23:28.498375+0000","last_became_active":"2024-03-19T01:23:26.511439+0000","last_became_peered":"2024-03-19T01:23:26.511439+0000","last_unstale":"2024-03-19T01:23:28.498375+0000","last_undegraded":"2024-03-19T01:23:28.498375+0000","last_fullsized":"2024-03-19T01:23:28.498375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569314+0000","last_change":"2024-03-19T01:23:27.523071+0000","last_active":"2024-03-19T01:23:28.569314+0000","last_peered":"2024-03-19T01:23:28.569314+0000","last_clean":"2024-03-19T01:23:28.569314+0000","last_became_active":"2024-03-19T01:23:27.522721+0000","last_became_peered":"2024-03-19T01:23:27.522721+0000","last_unstale":"2024-03-19T01:23:28.569314+0000","last_undegraded":"2024-03-19T01:23:28.569314+0000","last_fullsized":"2024-03-19T01:23:28.569314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556257+0000","last_change":"2024-03-19T01:23:26.543568+0000","last_active":"2024-03-19T01:23:27.556257+0000","last_peered":"2024-03-19T01:23:27.556257+0000","last_clean":"2024-03-19T01:23:27.556257+0000","last_became_active":"2024-03-19T01:23:26.542859+0000","last_became_peered":"2024-03-19T01:23:26.542859+0000","last_unstale":"2024-03-19T01:23:27.556257+0000","last_undegraded":"2024-03-19T01:23:27.556257+0000","last_fullsized":"2024-03-19T01:23:27.556257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:42.729108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.551540+0000","last_change":"2024-03-19T01:23:27.547084+0000","last_active":"2024-03-19T01:23:27.551540+0000","last_peered":"2024-03-19T01:23:27.551540+0000","last_clean":"2024-03-19T01:23:27.551540+0000","last_became_active":"2024-03-19T01:23:27.532180+0000","last_became_peered":"2024-03-19T01:23:27.532180+0000","last_unstale":"2024-03-19T01:23:27.551540+0000","last_undegraded":"2024-03-19T01:23:27.551540+0000","last_fullsized":"2024-03-19T01:23:27.551540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499777+0000","last_change":"2024-03-19T01:23:27.557382+0000","last_active":"2024-03-19T01:23:28.499777+0000","last_peered":"2024-03-19T01:23:28.499777+0000","last_clean":"2024-03-19T01:23:28.499777+0000","last_became_active":"2024-03-19T01:23:27.556485+0000","last_became_peered":"2024-03-19T01:23:27.556485+0000","last_unstale":"2024-03-19T01:23:28.499777+0000","last_undegraded":"2024-03-19T01:23:28.499777+0000","last_fullsized":"2024-03-19T01:23:28.499777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:01.486458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.563879+0000","last_change":"2024-03-19T01:23:26.541210+0000","last_active":"2024-03-19T01:23:27.563879+0000","last_peered":"2024-03-19T01:23:27.563879+0000","last_clean":"2024-03-19T01:23:27.563879+0000","last_became_active":"2024-03-19T01:23:26.540535+0000","last_became_peered":"2024-03-19T01:23:26.540535+0000","last_unstale":"2024-03-19T01:23:27.563879+0000","last_undegraded":"2024-03-19T01:23:27.563879+0000","last_fullsized":"2024-03-19T01:23:27.563879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:07.823972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499107+0000","last_change":"2024-03-19T01:23:27.547119+0000","last_active":"2024-03-19T01:23:28.499107+0000","last_peered":"2024-03-19T01:23:28.499107+0000","last_clean":"2024-03-19T01:23:28.499107+0000","last_became_active":"2024-03-19T01:23:27.537831+0000","last_became_peered":"2024-03-19T01:23:27.537831+0000","last_unstale":"2024-03-19T01:23:28.499107+0000","last_undegraded":"2024-03-19T01:23:28.499107+0000","last_fullsized":"2024-03-19T01:23:28.499107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.551073+0000","last_change":"2024-03-19T01:23:26.544693+0000","last_active":"2024-03-19T01:23:27.551073+0000","last_peered":"2024-03-19T01:23:27.551073+0000","last_clean":"2024-03-19T01:23:27.551073+0000","last_became_active":"2024-03-19T01:23:26.544047+0000","last_became_peered":"2024-03-19T01:23:26.544047+0000","last_unstale":"2024-03-19T01:23:27.551073+0000","last_undegraded":"2024-03-19T01:23:27.551073+0000","last_fullsized":"2024-03-19T01:23:27.551073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:36.870147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.550564+0000","last_change":"2024-03-19T01:23:27.549737+0000","last_active":"2024-03-19T01:23:27.550564+0000","last_peered":"2024-03-19T01:23:27.550564+0000","last_clean":"2024-03-19T01:23:27.550564+0000","last_became_active":"2024-03-19T01:23:27.549460+0000","last_became_peered":"2024-03-19T01:23:27.549460+0000","last_unstale":"2024-03-19T01:23:27.550564+0000","last_undegraded":"2024-03-19T01:23:27.550564+0000","last_fullsized":"2024-03-19T01:23:27.550564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:00.911501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569507+0000","last_change":"2024-03-19T01:23:26.520272+0000","last_active":"2024-03-19T01:23:28.569507+0000","last_peered":"2024-03-19T01:23:28.569507+0000","last_clean":"2024-03-19T01:23:28.569507+0000","last_became_active":"2024-03-19T01:23:26.519940+0000","last_became_peered":"2024-03-19T01:23:26.519940+0000","last_unstale":"2024-03-19T01:23:28.569507+0000","last_undegraded":"2024-03-19T01:23:28.569507+0000","last_fullsized":"2024-03-19T01:23:28.569507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:10.187739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569312+0000","last_change":"2024-03-19T01:23:27.541353+0000","last_active":"2024-03-19T01:23:28.569312+0000","last_peered":"2024-03-19T01:23:28.569312+0000","last_clean":"2024-03-19T01:23:28.569312+0000","last_became_active":"2024-03-19T01:23:27.541054+0000","last_became_peered":"2024-03-19T01:23:27.541054+0000","last_unstale":"2024-03-19T01:23:28.569312+0000","last_undegraded":"2024-03-19T01:23:28.569312+0000","last_fullsized":"2024-03-19T01:23:28.569312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556949+0000","last_change":"2024-03-19T01:23:26.528906+0000","last_active":"2024-03-19T01:23:27.556949+0000","last_peered":"2024-03-19T01:23:27.556949+0000","last_clean":"2024-03-19T01:23:27.556949+0000","last_became_active":"2024-03-19T01:23:26.528592+0000","last_became_peered":"2024-03-19T01:23:26.528592+0000","last_unstale":"2024-03-19T01:23:27.556949+0000","last_undegraded":"2024-03-19T01:23:27.556949+0000","last_fullsized":"2024-03-19T01:23:27.556949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":660255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":69431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":212992,"data_stored":69465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":660311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:23:31.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:23:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.940+0000 7f5d318eb640 1 Processor -- start 2024-03-19T01:23:31.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.940+0000 7f5d318eb640 1 -- start start 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d2c0b56e0 0x7f5d2c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5d2c072bf0 0x7f5d2c06b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d2c1488d0 con 0x7f5d2c06db00 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f5d2c06ee70 con 0x7f5d2c072bf0 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5d2c0b5ac0 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d2b7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5d2c06db00 0x7f5d2c06b800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53146/0 (socket says 172.21.15.7:53146) 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/3482267553 learned_addr learned my addr 172.21.15.7:0/3482267553 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d2a7fc640 1 --2- 172.21.15.7:0/3482267553 >> v2:172.21.15.7:3300/0 conn(0x7f5d2c072bf0 0x7f5d2c06b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d2affd640 1 --2- 172.21.15.7:0/3482267553 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d2c0b56e0 0x7f5d2c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d29ffb640 1 -- 172.21.15.7:0/3482267553 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1177659654 0 0) 0x7f5d2c1488d0 con 0x7f5d2c06db00 2024-03-19T01:23:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d29ffb640 1 -- 172.21.15.7:0/3482267553 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d14003530 con 0x7f5d2c06db00 2024-03-19T01:23:31.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d2a7fc640 1 -- 172.21.15.7:0/3482267553 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d2c0b56e0 msgr2=0x7f5d2c06d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:31.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d2a7fc640 1 --2- 172.21.15.7:0/3482267553 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d2c0b56e0 0x7f5d2c06d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d2a7fc640 1 -- 172.21.15.7:0/3482267553 >> v1:172.21.15.7:6789/0 conn(0x7f5d2c06db00 legacy=0x7f5d2c06b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d2a7fc640 1 -- 172.21.15.7:0/3482267553 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2c071120 con 0x7f5d2c072bf0 2024-03-19T01:23:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d2a7fc640 1 --2- 172.21.15.7:0/3482267553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c072bf0 0x7f5d2c06b2c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c06eff0 tx=0x7f5d1c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78074cabee59f115 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d29ffb640 1 -- 172.21.15.7:0/3482267553 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c0b0070 con 0x7f5d2c072bf0 2024-03-19T01:23:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d29ffb640 1 -- 172.21.15.7:0/3482267553 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d1c0ab600 con 0x7f5d2c072bf0 2024-03-19T01:23:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d29ffb640 1 -- 172.21.15.7:0/3482267553 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c0ab9d0 con 0x7f5d2c072bf0 2024-03-19T01:23:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 -- 172.21.15.7:0/3482267553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c072bf0 msgr2=0x7f5d2c06b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 --2- 172.21.15.7:0/3482267553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c072bf0 0x7f5d2c06b2c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c06eff0 tx=0x7f5d1c01c380 comp rx=0 tx=0).stop 2024-03-19T01:23:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 -- 172.21.15.7:0/3482267553 shutdown_connections 2024-03-19T01:23:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 --2- 172.21.15.7:0/3482267553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c072bf0 0x7f5d2c06b2c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 --2- 172.21.15.7:0/3482267553 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d2c0b56e0 0x7f5d2c06d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.944+0000 7f5d318eb640 1 -- 172.21.15.7:0/3482267553 >> 172.21.15.7:0/3482267553 conn(0x7f5d2c0b6510 msgr2=0x7f5d2c0b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 -- 172.21.15.7:0/3482267553 shutdown_connections 2024-03-19T01:23:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 -- 172.21.15.7:0/3482267553 wait complete. 2024-03-19T01:23:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 Processor -- start 2024-03-19T01:23:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 -- start start 2024-03-19T01:23:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d2c06db00 0x7f5d2c13fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d2c072bf0 0x7f5d2c1e0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c0b56e0 0x7f5d2c1e1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c06c090 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c06bf10 con 0x7f5d2c072bf0 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5d2c152420 con 0x7f5d2c06db00 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d2b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c0b56e0 0x7f5d2c1e1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d2b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c0b56e0 0x7f5d2c1e1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43160/0 (socket says 172.21.15.7:43160) 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/2811125618 learned_addr learned my addr 172.21.15.7:0/2811125618 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/2811125618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d2c06db00 msgr2=0x7f5d2c13fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d2b7fe640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d2c06db00 0x7f5d2c13fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/2811125618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d2c072bf0 msgr2=0x7f5d2c1e0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d2b7fe640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d2c072bf0 0x7f5d2c1e0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/2811125618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2c145d90 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d2b7fe640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c0b56e0 0x7f5d2c1e1290 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400cbb0 tx=0x7f5d2400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d13fff640 1 -- 172.21.15.7:0/2811125618 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24017070 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d13fff640 1 -- 172.21.15.7:0/2811125618 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d2400ad40 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2c146070 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d13fff640 1 -- 172.21.15.7:0/2811125618 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24012600 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2c1449a0 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2c075460 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.948+0000 7f5d13fff640 1 -- 172.21.15.7:0/2811125618 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5d2401a070 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.952+0000 7f5d13fff640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5d18067660 0x7f5d18069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:31.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.952+0000 7f5d2affd640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5d18067660 0x7f5d18069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:31.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.952+0000 7f5d13fff640 1 -- 172.21.15.7:0/2811125618 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5d24085df0 con 0x7f5d2c0b56e0 2024-03-19T01:23:31.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.952+0000 7f5d2affd640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5d18067660 0x7f5d18069b20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d20006aa0 tx=0x7f5d20006a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:31.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:31.956+0000 7f5d13fff640 1 -- 172.21.15.7:0/2811125618 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5d24056490 con 0x7f5d2c0b56e0 2024-03-19T01:23:32.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.104+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5d2c0694b0 con 0x7f5d18067660 2024-03-19T01:23:32.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.112+0000 7f5d13fff640 1 -- 172.21.15.7:0/2811125618 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251922 (secure 0 0 0) 0x7f5d2c0694b0 con 0x7f5d18067660 2024-03-19T01:23:32.117 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:32.118 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.112+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5d18067660 msgr2=0x7f5d18069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.112+0000 7f5d318eb640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5d18067660 0x7f5d18069b20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d20006aa0 tx=0x7f5d20006a30 comp rx=0 tx=0).stop 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.112+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c0b56e0 msgr2=0x7f5d2c1e1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.112+0000 7f5d318eb640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c0b56e0 0x7f5d2c1e1290 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400cbb0 tx=0x7f5d2400a430 comp rx=0 tx=0).stop 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.116+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 shutdown_connections 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.116+0000 7f5d318eb640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5d18067660 0x7f5d18069b20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.116+0000 7f5d318eb640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d2c0b56e0 0x7f5d2c1e1290 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.116+0000 7f5d318eb640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d2c072bf0 0x7f5d2c1e0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.116+0000 7f5d318eb640 1 --2- 172.21.15.7:0/2811125618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d2c06db00 0x7f5d2c13fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.116+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 >> 172.21.15.7:0/2811125618 conn(0x7f5d2c0b6510 msgr2=0x7f5d2c071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.116+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 shutdown_connections 2024-03-19T01:23:32.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.116+0000 7f5d318eb640 1 -- 172.21.15.7:0/2811125618 wait complete. 2024-03-19T01:23:32.133 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-19T01:23:30.890859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235652,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739068,"statfs":{"total":773094113280,"available":772852805632,"internally_reserved":0,"allocated":2891776,"data_stored":1739996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.413785"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.559151+0000","last_change":"2024-03-19T01:23:27.551581+0000","last_active":"2024-03-19T01:23:27.559151+0000","last_peered":"2024-03-19T01:23:27.559151+0000","last_clean":"2024-03-19T01:23:27.559151+0000","last_became_active":"2024-03-19T01:23:27.550909+0000","last_became_peered":"2024-03-19T01:23:27.550909+0000","last_unstale":"2024-03-19T01:23:27.559151+0000","last_undegraded":"2024-03-19T01:23:27.559151+0000","last_fullsized":"2024-03-19T01:23:27.559151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498885+0000","last_change":"2024-03-19T01:23:27.512582+0000","last_active":"2024-03-19T01:23:28.498885+0000","last_peered":"2024-03-19T01:23:28.498885+0000","last_clean":"2024-03-19T01:23:28.498885+0000","last_became_active":"2024-03-19T01:23:27.512252+0000","last_became_peered":"2024-03-19T01:23:27.512252+0000","last_unstale":"2024-03-19T01:23:28.498885+0000","last_undegraded":"2024-03-19T01:23:28.498885+0000","last_fullsized":"2024-03-19T01:23:28.498885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.514378+0000","last_change":"2024-03-19T01:23:27.512676+0000","last_active":"2024-03-19T01:23:27.514378+0000","last_peered":"2024-03-19T01:23:27.514378+0000","last_clean":"2024-03-19T01:23:27.514378+0000","last_became_active":"2024-03-19T01:23:27.512311+0000","last_became_peered":"2024-03-19T01:23:27.512311+0000","last_unstale":"2024-03-19T01:23:27.514378+0000","last_undegraded":"2024-03-19T01:23:27.514378+0000","last_fullsized":"2024-03-19T01:23:27.514378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:31.282958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498730+0000","last_change":"2024-03-19T01:23:27.538857+0000","last_active":"2024-03-19T01:23:28.498730+0000","last_peered":"2024-03-19T01:23:28.498730+0000","last_clean":"2024-03-19T01:23:28.498730+0000","last_became_active":"2024-03-19T01:23:27.536808+0000","last_became_peered":"2024-03-19T01:23:27.536808+0000","last_unstale":"2024-03-19T01:23:28.498730+0000","last_undegraded":"2024-03-19T01:23:28.498730+0000","last_fullsized":"2024-03-19T01:23:28.498730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556403+0000","last_change":"2024-03-19T01:23:27.543517+0000","last_active":"2024-03-19T01:23:27.556403+0000","last_peered":"2024-03-19T01:23:27.556403+0000","last_clean":"2024-03-19T01:23:27.556403+0000","last_became_active":"2024-03-19T01:23:27.542861+0000","last_became_peered":"2024-03-19T01:23:27.542861+0000","last_unstale":"2024-03-19T01:23:27.556403+0000","last_undegraded":"2024-03-19T01:23:27.556403+0000","last_fullsized":"2024-03-19T01:23:27.556403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:12:50.425783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498639+0000","last_change":"2024-03-19T01:23:27.549333+0000","last_active":"2024-03-19T01:23:28.498639+0000","last_peered":"2024-03-19T01:23:28.498639+0000","last_clean":"2024-03-19T01:23:28.498639+0000","last_became_active":"2024-03-19T01:23:27.548338+0000","last_became_peered":"2024-03-19T01:23:27.548338+0000","last_unstale":"2024-03-19T01:23:28.498639+0000","last_undegraded":"2024-03-19T01:23:28.498639+0000","last_fullsized":"2024-03-19T01:23:28.498639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556500+0000","last_change":"2024-03-19T01:23:27.536206+0000","last_active":"2024-03-19T01:23:27.556500+0000","last_peered":"2024-03-19T01:23:27.556500+0000","last_clean":"2024-03-19T01:23:27.556500+0000","last_became_active":"2024-03-19T01:23:27.535402+0000","last_became_peered":"2024-03-19T01:23:27.535402+0000","last_unstale":"2024-03-19T01:23:27.556500+0000","last_undegraded":"2024-03-19T01:23:27.556500+0000","last_fullsized":"2024-03-19T01:23:27.556500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:10.543122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569176+0000","last_change":"2024-03-19T01:23:27.548444+0000","last_active":"2024-03-19T01:23:28.569176+0000","last_peered":"2024-03-19T01:23:28.569176+0000","last_clean":"2024-03-19T01:23:28.569176+0000","last_became_active":"2024-03-19T01:23:27.548160+0000","last_became_peered":"2024-03-19T01:23:27.548160+0000","last_unstale":"2024-03-19T01:23:28.569176+0000","last_undegraded":"2024-03-19T01:23:28.569176+0000","last_fullsized":"2024-03-19T01:23:28.569176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.560428+0000","last_change":"2024-03-19T01:23:27.558139+0000","last_active":"2024-03-19T01:23:27.560428+0000","last_peered":"2024-03-19T01:23:27.560428+0000","last_clean":"2024-03-19T01:23:27.560428+0000","last_became_active":"2024-03-19T01:23:27.557564+0000","last_became_peered":"2024-03-19T01:23:27.557564+0000","last_unstale":"2024-03-19T01:23:27.560428+0000","last_undegraded":"2024-03-19T01:23:27.560428+0000","last_fullsized":"2024-03-19T01:23:27.560428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.552973+0000","last_change":"2024-03-19T01:23:27.549847+0000","last_active":"2024-03-19T01:23:27.552973+0000","last_peered":"2024-03-19T01:23:27.552973+0000","last_clean":"2024-03-19T01:23:27.552973+0000","last_became_active":"2024-03-19T01:23:27.548601+0000","last_became_peered":"2024-03-19T01:23:27.548601+0000","last_unstale":"2024-03-19T01:23:27.552973+0000","last_undegraded":"2024-03-19T01:23:27.552973+0000","last_fullsized":"2024-03-19T01:23:27.552973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:23.802985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.560073+0000","last_change":"2024-03-19T01:23:27.558581+0000","last_active":"2024-03-19T01:23:27.560073+0000","last_peered":"2024-03-19T01:23:27.560073+0000","last_clean":"2024-03-19T01:23:27.560073+0000","last_became_active":"2024-03-19T01:23:27.558044+0000","last_became_peered":"2024-03-19T01:23:27.558044+0000","last_unstale":"2024-03-19T01:23:27.560073+0000","last_undegraded":"2024-03-19T01:23:27.560073+0000","last_fullsized":"2024-03-19T01:23:27.560073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:10.543122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831030+0000","last_change":"2024-03-19T01:23:27.543968+0000","last_active":"2024-03-19T01:23:28.831030+0000","last_peered":"2024-03-19T01:23:28.831030+0000","last_clean":"2024-03-19T01:23:28.831030+0000","last_became_active":"2024-03-19T01:23:27.543453+0000","last_became_peered":"2024-03-19T01:23:27.543453+0000","last_unstale":"2024-03-19T01:23:28.831030+0000","last_undegraded":"2024-03-19T01:23:28.831030+0000","last_fullsized":"2024-03-19T01:23:28.831030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569582+0000","last_change":"2024-03-19T01:23:27.541445+0000","last_active":"2024-03-19T01:23:28.569582+0000","last_peered":"2024-03-19T01:23:28.569582+0000","last_clean":"2024-03-19T01:23:28.569582+0000","last_became_active":"2024-03-19T01:23:27.541162+0000","last_became_peered":"2024-03-19T01:23:27.541162+0000","last_unstale":"2024-03-19T01:23:28.569582+0000","last_undegraded":"2024-03-19T01:23:28.569582+0000","last_fullsized":"2024-03-19T01:23:28.569582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.516422+0000","last_change":"2024-03-19T01:23:27.515384+0000","last_active":"2024-03-19T01:23:27.516422+0000","last_peered":"2024-03-19T01:23:27.516422+0000","last_clean":"2024-03-19T01:23:27.516422+0000","last_became_active":"2024-03-19T01:23:27.514935+0000","last_became_peered":"2024-03-19T01:23:27.514935+0000","last_unstale":"2024-03-19T01:23:27.516422+0000","last_undegraded":"2024-03-19T01:23:27.516422+0000","last_fullsized":"2024-03-19T01:23:27.516422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:54.910728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497668+0000","last_change":"2024-03-19T01:23:27.509553+0000","last_active":"2024-03-19T01:23:28.497668+0000","last_peered":"2024-03-19T01:23:28.497668+0000","last_clean":"2024-03-19T01:23:28.497668+0000","last_became_active":"2024-03-19T01:23:27.509178+0000","last_became_peered":"2024-03-19T01:23:27.509178+0000","last_unstale":"2024-03-19T01:23:28.497668+0000","last_undegraded":"2024-03-19T01:23:28.497668+0000","last_fullsized":"2024-03-19T01:23:28.497668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498214+0000","last_change":"2024-03-19T01:23:27.547056+0000","last_active":"2024-03-19T01:23:28.498214+0000","last_peered":"2024-03-19T01:23:28.498214+0000","last_clean":"2024-03-19T01:23:28.498214+0000","last_became_active":"2024-03-19T01:23:27.538339+0000","last_became_peered":"2024-03-19T01:23:27.538339+0000","last_unstale":"2024-03-19T01:23:28.498214+0000","last_undegraded":"2024-03-19T01:23:28.498214+0000","last_fullsized":"2024-03-19T01:23:28.498214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.500126+0000","last_change":"2024-03-19T01:23:27.562551+0000","last_active":"2024-03-19T01:23:28.500126+0000","last_peered":"2024-03-19T01:23:28.500126+0000","last_clean":"2024-03-19T01:23:28.500126+0000","last_became_active":"2024-03-19T01:23:27.561996+0000","last_became_peered":"2024-03-19T01:23:27.561996+0000","last_unstale":"2024-03-19T01:23:28.500126+0000","last_undegraded":"2024-03-19T01:23:28.500126+0000","last_fullsized":"2024-03-19T01:23:28.500126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:27.331497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572895+0000","last_change":"2024-03-19T01:23:27.541038+0000","last_active":"2024-03-19T01:23:28.572895+0000","last_peered":"2024-03-19T01:23:28.572895+0000","last_clean":"2024-03-19T01:23:28.572895+0000","last_became_active":"2024-03-19T01:23:27.540751+0000","last_became_peered":"2024-03-19T01:23:27.540751+0000","last_unstale":"2024-03-19T01:23:28.572895+0000","last_undegraded":"2024-03-19T01:23:28.572895+0000","last_fullsized":"2024-03-19T01:23:28.572895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.559025+0000","last_change":"2024-03-19T01:23:27.552053+0000","last_active":"2024-03-19T01:23:27.559025+0000","last_peered":"2024-03-19T01:23:27.559025+0000","last_clean":"2024-03-19T01:23:27.559025+0000","last_became_active":"2024-03-19T01:23:27.551405+0000","last_became_peered":"2024-03-19T01:23:27.551405+0000","last_unstale":"2024-03-19T01:23:27.559025+0000","last_undegraded":"2024-03-19T01:23:27.559025+0000","last_fullsized":"2024-03-19T01:23:27.559025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.553016+0000","last_change":"2024-03-19T01:23:27.548149+0000","last_active":"2024-03-19T01:23:27.553016+0000","last_peered":"2024-03-19T01:23:27.553016+0000","last_clean":"2024-03-19T01:23:27.553016+0000","last_became_active":"2024-03-19T01:23:27.542105+0000","last_became_peered":"2024-03-19T01:23:27.542105+0000","last_unstale":"2024-03-19T01:23:27.553016+0000","last_undegraded":"2024-03-19T01:23:27.553016+0000","last_fullsized":"2024-03-19T01:23:27.553016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:00.911501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.523235+0000","last_change":"2024-03-19T01:23:27.521814+0000","last_active":"2024-03-19T01:23:27.523235+0000","last_peered":"2024-03-19T01:23:27.523235+0000","last_clean":"2024-03-19T01:23:27.523235+0000","last_became_active":"2024-03-19T01:23:27.519639+0000","last_became_peered":"2024-03-19T01:23:27.519639+0000","last_unstale":"2024-03-19T01:23:27.523235+0000","last_undegraded":"2024-03-19T01:23:27.523235+0000","last_fullsized":"2024-03-19T01:23:27.523235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:08.835543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499387+0000","last_change":"2024-03-19T01:23:27.558248+0000","last_active":"2024-03-19T01:23:28.499387+0000","last_peered":"2024-03-19T01:23:28.499387+0000","last_clean":"2024-03-19T01:23:28.499387+0000","last_became_active":"2024-03-19T01:23:27.557663+0000","last_became_peered":"2024-03-19T01:23:27.557663+0000","last_unstale":"2024-03-19T01:23:28.499387+0000","last_undegraded":"2024-03-19T01:23:28.499387+0000","last_fullsized":"2024-03-19T01:23:28.499387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.568928+0000","last_change":"2024-03-19T01:23:27.532119+0000","last_active":"2024-03-19T01:23:28.568928+0000","last_peered":"2024-03-19T01:23:28.568928+0000","last_clean":"2024-03-19T01:23:28.568928+0000","last_became_active":"2024-03-19T01:23:27.531846+0000","last_became_peered":"2024-03-19T01:23:27.531846+0000","last_unstale":"2024-03-19T01:23:28.568928+0000","last_undegraded":"2024-03-19T01:23:28.568928+0000","last_fullsized":"2024-03-19T01:23:28.568928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498373+0000","last_change":"2024-03-19T01:23:27.546292+0000","last_active":"2024-03-19T01:23:28.498373+0000","last_peered":"2024-03-19T01:23:28.498373+0000","last_clean":"2024-03-19T01:23:28.498373+0000","last_became_active":"2024-03-19T01:23:27.537369+0000","last_became_peered":"2024-03-19T01:23:27.537369+0000","last_unstale":"2024-03-19T01:23:28.498373+0000","last_undegraded":"2024-03-19T01:23:28.498373+0000","last_fullsized":"2024-03-19T01:23:28.498373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569601+0000","last_change":"2024-03-19T01:23:27.522448+0000","last_active":"2024-03-19T01:23:28.569601+0000","last_peered":"2024-03-19T01:23:28.569601+0000","last_clean":"2024-03-19T01:23:28.569601+0000","last_became_active":"2024-03-19T01:23:27.522201+0000","last_became_peered":"2024-03-19T01:23:27.522201+0000","last_unstale":"2024-03-19T01:23:28.569601+0000","last_undegraded":"2024-03-19T01:23:28.569601+0000","last_fullsized":"2024-03-19T01:23:28.569601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.553016+0000","last_change":"2024-03-19T01:23:27.551178+0000","last_active":"2024-03-19T01:23:27.553016+0000","last_peered":"2024-03-19T01:23:27.553016+0000","last_clean":"2024-03-19T01:23:27.553016+0000","last_became_active":"2024-03-19T01:23:27.550571+0000","last_became_peered":"2024-03-19T01:23:27.550571+0000","last_unstale":"2024-03-19T01:23:27.553016+0000","last_undegraded":"2024-03-19T01:23:27.553016+0000","last_fullsized":"2024-03-19T01:23:27.553016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:52.938759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497894+0000","last_change":"2024-03-19T01:23:27.509194+0000","last_active":"2024-03-19T01:23:28.497894+0000","last_peered":"2024-03-19T01:23:28.497894+0000","last_clean":"2024-03-19T01:23:28.497894+0000","last_became_active":"2024-03-19T01:23:27.508796+0000","last_became_peered":"2024-03-19T01:23:27.508796+0000","last_unstale":"2024-03-19T01:23:28.497894+0000","last_undegraded":"2024-03-19T01:23:28.497894+0000","last_fullsized":"2024-03-19T01:23:28.497894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569258+0000","last_change":"2024-03-19T01:23:27.522380+0000","last_active":"2024-03-19T01:23:28.569258+0000","last_peered":"2024-03-19T01:23:28.569258+0000","last_clean":"2024-03-19T01:23:28.569258+0000","last_became_active":"2024-03-19T01:23:27.521968+0000","last_became_peered":"2024-03-19T01:23:27.521968+0000","last_unstale":"2024-03-19T01:23:28.569258+0000","last_undegraded":"2024-03-19T01:23:28.569258+0000","last_fullsized":"2024-03-19T01:23:28.569258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.519923+0000","last_change":"2024-03-19T01:23:27.516056+0000","last_active":"2024-03-19T01:23:27.519923+0000","last_peered":"2024-03-19T01:23:27.519923+0000","last_clean":"2024-03-19T01:23:27.519923+0000","last_became_active":"2024-03-19T01:23:27.515651+0000","last_became_peered":"2024-03-19T01:23:27.515651+0000","last_unstale":"2024-03-19T01:23:27.519923+0000","last_undegraded":"2024-03-19T01:23:27.519923+0000","last_fullsized":"2024-03-19T01:23:27.519923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:38.911150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499052+0000","last_change":"2024-03-19T01:23:27.547751+0000","last_active":"2024-03-19T01:23:28.499052+0000","last_peered":"2024-03-19T01:23:28.499052+0000","last_clean":"2024-03-19T01:23:28.499052+0000","last_became_active":"2024-03-19T01:23:27.530027+0000","last_became_peered":"2024-03-19T01:23:27.530027+0000","last_unstale":"2024-03-19T01:23:28.499052+0000","last_undegraded":"2024-03-19T01:23:28.499052+0000","last_fullsized":"2024-03-19T01:23:28.499052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572767+0000","last_change":"2024-03-19T01:23:27.538411+0000","last_active":"2024-03-19T01:23:28.572767+0000","last_peered":"2024-03-19T01:23:28.572767+0000","last_clean":"2024-03-19T01:23:28.572767+0000","last_became_active":"2024-03-19T01:23:27.537893+0000","last_became_peered":"2024-03-19T01:23:27.537893+0000","last_unstale":"2024-03-19T01:23:28.572767+0000","last_undegraded":"2024-03-19T01:23:28.572767+0000","last_fullsized":"2024-03-19T01:23:28.572767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.548480+0000","last_change":"2024-03-19T01:23:27.539735+0000","last_active":"2024-03-19T01:23:27.548480+0000","last_peered":"2024-03-19T01:23:27.548480+0000","last_clean":"2024-03-19T01:23:27.548480+0000","last_became_active":"2024-03-19T01:23:27.539022+0000","last_became_peered":"2024-03-19T01:23:27.539022+0000","last_unstale":"2024-03-19T01:23:27.548480+0000","last_undegraded":"2024-03-19T01:23:27.548480+0000","last_fullsized":"2024-03-19T01:23:27.548480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:08.835543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830822+0000","last_change":"2024-03-19T01:23:27.503767+0000","last_active":"2024-03-19T01:23:28.830822+0000","last_peered":"2024-03-19T01:23:28.830822+0000","last_clean":"2024-03-19T01:23:28.830822+0000","last_became_active":"2024-03-19T01:23:27.503030+0000","last_became_peered":"2024-03-19T01:23:27.503030+0000","last_unstale":"2024-03-19T01:23:28.830822+0000","last_undegraded":"2024-03-19T01:23:28.830822+0000","last_fullsized":"2024-03-19T01:23:28.830822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:08.835543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569515+0000","last_change":"2024-03-19T01:23:26.532832+0000","last_active":"2024-03-19T01:23:28.569515+0000","last_peered":"2024-03-19T01:23:28.569515+0000","last_clean":"2024-03-19T01:23:28.569515+0000","last_became_active":"2024-03-19T01:23:26.532516+0000","last_became_peered":"2024-03-19T01:23:26.532516+0000","last_unstale":"2024-03-19T01:23:28.569515+0000","last_undegraded":"2024-03-19T01:23:28.569515+0000","last_fullsized":"2024-03-19T01:23:28.569515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.523812+0000","last_change":"2024-03-19T01:23:27.522605+0000","last_active":"2024-03-19T01:23:27.523812+0000","last_peered":"2024-03-19T01:23:27.523812+0000","last_clean":"2024-03-19T01:23:27.523812+0000","last_became_active":"2024-03-19T01:23:27.522344+0000","last_became_peered":"2024-03-19T01:23:27.522344+0000","last_unstale":"2024-03-19T01:23:27.523812+0000","last_undegraded":"2024-03-19T01:23:27.523812+0000","last_fullsized":"2024-03-19T01:23:27.523812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.547716+0000","last_change":"2024-03-19T01:23:26.535891+0000","last_active":"2024-03-19T01:23:27.547716+0000","last_peered":"2024-03-19T01:23:27.547716+0000","last_clean":"2024-03-19T01:23:27.547716+0000","last_became_active":"2024-03-19T01:23:26.535218+0000","last_became_peered":"2024-03-19T01:23:26.535218+0000","last_unstale":"2024-03-19T01:23:27.547716+0000","last_undegraded":"2024-03-19T01:23:27.547716+0000","last_fullsized":"2024-03-19T01:23:27.547716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.568845+0000","last_change":"2024-03-19T01:23:26.536647+0000","last_active":"2024-03-19T01:23:28.568845+0000","last_peered":"2024-03-19T01:23:28.568845+0000","last_clean":"2024-03-19T01:23:28.568845+0000","last_became_active":"2024-03-19T01:23:26.535919+0000","last_became_peered":"2024-03-19T01:23:26.535919+0000","last_unstale":"2024-03-19T01:23:28.568845+0000","last_undegraded":"2024-03-19T01:23:28.568845+0000","last_fullsized":"2024-03-19T01:23:28.568845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569044+0000","last_change":"2024-03-19T01:23:27.517889+0000","last_active":"2024-03-19T01:23:28.569044+0000","last_peered":"2024-03-19T01:23:28.569044+0000","last_clean":"2024-03-19T01:23:28.569044+0000","last_became_active":"2024-03-19T01:23:27.515221+0000","last_became_peered":"2024-03-19T01:23:27.515221+0000","last_unstale":"2024-03-19T01:23:28.569044+0000","last_undegraded":"2024-03-19T01:23:28.569044+0000","last_fullsized":"2024-03-19T01:23:28.569044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499348+0000","last_change":"2024-03-19T01:23:26.518817+0000","last_active":"2024-03-19T01:23:28.499348+0000","last_peered":"2024-03-19T01:23:28.499348+0000","last_clean":"2024-03-19T01:23:28.499348+0000","last_became_active":"2024-03-19T01:23:26.518008+0000","last_became_peered":"2024-03-19T01:23:26.518008+0000","last_unstale":"2024-03-19T01:23:28.499348+0000","last_undegraded":"2024-03-19T01:23:28.499348+0000","last_fullsized":"2024-03-19T01:23:28.499348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498152+0000","last_change":"2024-03-19T01:23:27.514550+0000","last_active":"2024-03-19T01:23:28.498152+0000","last_peered":"2024-03-19T01:23:28.498152+0000","last_clean":"2024-03-19T01:23:28.498152+0000","last_became_active":"2024-03-19T01:23:27.512624+0000","last_became_peered":"2024-03-19T01:23:27.512624+0000","last_unstale":"2024-03-19T01:23:28.498152+0000","last_undegraded":"2024-03-19T01:23:28.498152+0000","last_fullsized":"2024-03-19T01:23:28.498152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.503744+0000","last_change":"2024-03-19T01:23:26.536461+0000","last_active":"2024-03-19T01:23:27.503744+0000","last_peered":"2024-03-19T01:23:27.503744+0000","last_clean":"2024-03-19T01:23:27.503744+0000","last_became_active":"2024-03-19T01:23:26.535707+0000","last_became_peered":"2024-03-19T01:23:26.535707+0000","last_unstale":"2024-03-19T01:23:27.503744+0000","last_undegraded":"2024-03-19T01:23:27.503744+0000","last_fullsized":"2024-03-19T01:23:27.503744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499599+0000","last_change":"2024-03-19T01:23:27.561680+0000","last_active":"2024-03-19T01:23:28.499599+0000","last_peered":"2024-03-19T01:23:28.499599+0000","last_clean":"2024-03-19T01:23:28.499599+0000","last_became_active":"2024-03-19T01:23:27.559667+0000","last_became_peered":"2024-03-19T01:23:27.559667+0000","last_unstale":"2024-03-19T01:23:28.499599+0000","last_undegraded":"2024-03-19T01:23:28.499599+0000","last_fullsized":"2024-03-19T01:23:28.499599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:28.568774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499535+0000","last_change":"2024-03-19T01:23:26.541115+0000","last_active":"2024-03-19T01:23:28.499535+0000","last_peered":"2024-03-19T01:23:28.499535+0000","last_clean":"2024-03-19T01:23:28.499535+0000","last_became_active":"2024-03-19T01:23:26.540408+0000","last_became_peered":"2024-03-19T01:23:26.540408+0000","last_unstale":"2024-03-19T01:23:28.499535+0000","last_undegraded":"2024-03-19T01:23:28.499535+0000","last_fullsized":"2024-03-19T01:23:28.499535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.558153+0000","last_change":"2024-03-19T01:23:27.557631+0000","last_active":"2024-03-19T01:23:27.558153+0000","last_peered":"2024-03-19T01:23:27.558153+0000","last_clean":"2024-03-19T01:23:27.558153+0000","last_became_active":"2024-03-19T01:23:27.557324+0000","last_became_peered":"2024-03-19T01:23:27.557324+0000","last_unstale":"2024-03-19T01:23:27.558153+0000","last_undegraded":"2024-03-19T01:23:27.558153+0000","last_fullsized":"2024-03-19T01:23:27.558153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:23.802985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498202+0000","last_change":"2024-03-19T01:23:26.517007+0000","last_active":"2024-03-19T01:23:28.498202+0000","last_peered":"2024-03-19T01:23:28.498202+0000","last_clean":"2024-03-19T01:23:28.498202+0000","last_became_active":"2024-03-19T01:23:26.516710+0000","last_became_peered":"2024-03-19T01:23:26.516710+0000","last_unstale":"2024-03-19T01:23:28.498202+0000","last_undegraded":"2024-03-19T01:23:28.498202+0000","last_fullsized":"2024-03-19T01:23:28.498202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:36.870147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831243+0000","last_change":"2024-03-19T01:23:27.515805+0000","last_active":"2024-03-19T01:23:28.831243+0000","last_peered":"2024-03-19T01:23:28.831243+0000","last_clean":"2024-03-19T01:23:28.831243+0000","last_became_active":"2024-03-19T01:23:27.515099+0000","last_became_peered":"2024-03-19T01:23:27.515099+0000","last_unstale":"2024-03-19T01:23:28.831243+0000","last_undegraded":"2024-03-19T01:23:28.831243+0000","last_fullsized":"2024-03-19T01:23:28.831243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:00.911501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.557475+0000","last_change":"2024-03-19T01:23:26.535383+0000","last_active":"2024-03-19T01:23:27.557475+0000","last_peered":"2024-03-19T01:23:27.557475+0000","last_clean":"2024-03-19T01:23:27.557475+0000","last_became_active":"2024-03-19T01:23:26.534052+0000","last_became_peered":"2024-03-19T01:23:26.534052+0000","last_unstale":"2024-03-19T01:23:27.557475+0000","last_undegraded":"2024-03-19T01:23:27.557475+0000","last_fullsized":"2024-03-19T01:23:27.557475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.558988+0000","last_change":"2024-03-19T01:23:27.550833+0000","last_active":"2024-03-19T01:23:27.558988+0000","last_peered":"2024-03-19T01:23:27.558988+0000","last_clean":"2024-03-19T01:23:27.558988+0000","last_became_active":"2024-03-19T01:23:27.544992+0000","last_became_peered":"2024-03-19T01:23:27.544992+0000","last_unstale":"2024-03-19T01:23:27.558988+0000","last_undegraded":"2024-03-19T01:23:27.558988+0000","last_fullsized":"2024-03-19T01:23:27.558988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.526141+0000","last_change":"2024-03-19T01:23:26.506460+0000","last_active":"2024-03-19T01:23:27.526141+0000","last_peered":"2024-03-19T01:23:27.526141+0000","last_clean":"2024-03-19T01:23:27.526141+0000","last_became_active":"2024-03-19T01:23:26.506155+0000","last_became_peered":"2024-03-19T01:23:26.506155+0000","last_unstale":"2024-03-19T01:23:27.526141+0000","last_undegraded":"2024-03-19T01:23:27.526141+0000","last_fullsized":"2024-03-19T01:23:27.526141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:10.187739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.515234+0000","last_change":"2024-03-19T01:23:27.514233+0000","last_active":"2024-03-19T01:23:27.515234+0000","last_peered":"2024-03-19T01:23:27.515234+0000","last_clean":"2024-03-19T01:23:27.515234+0000","last_became_active":"2024-03-19T01:23:27.513950+0000","last_became_peered":"2024-03-19T01:23:27.513950+0000","last_unstale":"2024-03-19T01:23:27.515234+0000","last_undegraded":"2024-03-19T01:23:27.515234+0000","last_fullsized":"2024-03-19T01:23:27.515234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:51.109900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499325+0000","last_change":"2024-03-19T01:23:26.516202+0000","last_active":"2024-03-19T01:23:28.499325+0000","last_peered":"2024-03-19T01:23:28.499325+0000","last_clean":"2024-03-19T01:23:28.499325+0000","last_became_active":"2024-03-19T01:23:26.515081+0000","last_became_peered":"2024-03-19T01:23:26.515081+0000","last_unstale":"2024-03-19T01:23:28.499325+0000","last_undegraded":"2024-03-19T01:23:28.499325+0000","last_fullsized":"2024-03-19T01:23:28.499325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.555944+0000","last_change":"2024-03-19T01:23:27.534410+0000","last_active":"2024-03-19T01:23:27.555944+0000","last_peered":"2024-03-19T01:23:27.555944+0000","last_clean":"2024-03-19T01:23:27.555944+0000","last_became_active":"2024-03-19T01:23:27.533583+0000","last_became_peered":"2024-03-19T01:23:27.533583+0000","last_unstale":"2024-03-19T01:23:27.555944+0000","last_undegraded":"2024-03-19T01:23:27.555944+0000","last_fullsized":"2024-03-19T01:23:27.555944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:23.802985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.505196+0000","last_change":"2024-03-19T01:23:26.530022+0000","last_active":"2024-03-19T01:23:27.505196+0000","last_peered":"2024-03-19T01:23:27.505196+0000","last_clean":"2024-03-19T01:23:27.505196+0000","last_became_active":"2024-03-19T01:23:26.525529+0000","last_became_peered":"2024-03-19T01:23:26.525529+0000","last_unstale":"2024-03-19T01:23:27.505196+0000","last_undegraded":"2024-03-19T01:23:27.505196+0000","last_fullsized":"2024-03-19T01:23:27.505196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.551502+0000","last_change":"2024-03-19T01:23:27.542939+0000","last_active":"2024-03-19T01:23:27.551502+0000","last_peered":"2024-03-19T01:23:27.551502+0000","last_clean":"2024-03-19T01:23:27.551502+0000","last_became_active":"2024-03-19T01:23:27.532364+0000","last_became_peered":"2024-03-19T01:23:27.532364+0000","last_unstale":"2024-03-19T01:23:27.551502+0000","last_undegraded":"2024-03-19T01:23:27.551502+0000","last_fullsized":"2024-03-19T01:23:27.551502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572800+0000","last_change":"2024-03-19T01:23:26.513589+0000","last_active":"2024-03-19T01:23:28.572800+0000","last_peered":"2024-03-19T01:23:28.572800+0000","last_clean":"2024-03-19T01:23:28.572800+0000","last_became_active":"2024-03-19T01:23:26.513223+0000","last_became_peered":"2024-03-19T01:23:26.513223+0000","last_unstale":"2024-03-19T01:23:28.572800+0000","last_undegraded":"2024-03-19T01:23:28.572800+0000","last_fullsized":"2024-03-19T01:23:28.572800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.558933+0000","last_change":"2024-03-19T01:23:27.552435+0000","last_active":"2024-03-19T01:23:27.558933+0000","last_peered":"2024-03-19T01:23:27.558933+0000","last_clean":"2024-03-19T01:23:27.558933+0000","last_became_active":"2024-03-19T01:23:27.545233+0000","last_became_peered":"2024-03-19T01:23:27.545233+0000","last_unstale":"2024-03-19T01:23:27.558933+0000","last_undegraded":"2024-03-19T01:23:27.558933+0000","last_fullsized":"2024-03-19T01:23:27.558933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831253+0000","last_change":"2024-03-19T01:23:26.515010+0000","last_active":"2024-03-19T01:23:28.831253+0000","last_peered":"2024-03-19T01:23:28.831253+0000","last_clean":"2024-03-19T01:23:28.831253+0000","last_became_active":"2024-03-19T01:23:26.513690+0000","last_became_peered":"2024-03-19T01:23:26.513690+0000","last_unstale":"2024-03-19T01:23:28.831253+0000","last_undegraded":"2024-03-19T01:23:28.831253+0000","last_fullsized":"2024-03-19T01:23:28.831253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:41:56.780781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556636+0000","last_change":"2024-03-19T01:23:27.535739+0000","last_active":"2024-03-19T01:23:27.556636+0000","last_peered":"2024-03-19T01:23:27.556636+0000","last_clean":"2024-03-19T01:23:27.556636+0000","last_became_active":"2024-03-19T01:23:27.534174+0000","last_became_peered":"2024-03-19T01:23:27.534174+0000","last_unstale":"2024-03-19T01:23:27.556636+0000","last_undegraded":"2024-03-19T01:23:27.556636+0000","last_fullsized":"2024-03-19T01:23:27.556636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:52.938759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572994+0000","last_change":"2024-03-19T01:23:26.512861+0000","last_active":"2024-03-19T01:23:28.572994+0000","last_peered":"2024-03-19T01:23:28.572994+0000","last_clean":"2024-03-19T01:23:28.572994+0000","last_became_active":"2024-03-19T01:23:26.512475+0000","last_became_peered":"2024-03-19T01:23:26.512475+0000","last_unstale":"2024-03-19T01:23:28.572994+0000","last_undegraded":"2024-03-19T01:23:28.572994+0000","last_fullsized":"2024-03-19T01:23:28.572994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.560392+0000","last_change":"2024-03-19T01:23:27.553761+0000","last_active":"2024-03-19T01:23:27.560392+0000","last_peered":"2024-03-19T01:23:27.560392+0000","last_clean":"2024-03-19T01:23:27.560392+0000","last_became_active":"2024-03-19T01:23:27.551510+0000","last_became_peered":"2024-03-19T01:23:27.551510+0000","last_unstale":"2024-03-19T01:23:27.560392+0000","last_undegraded":"2024-03-19T01:23:27.560392+0000","last_fullsized":"2024-03-19T01:23:27.560392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498228+0000","last_change":"2024-03-19T01:23:26.520079+0000","last_active":"2024-03-19T01:23:28.498228+0000","last_peered":"2024-03-19T01:23:28.498228+0000","last_clean":"2024-03-19T01:23:28.498228+0000","last_became_active":"2024-03-19T01:23:26.519678+0000","last_became_peered":"2024-03-19T01:23:26.519678+0000","last_unstale":"2024-03-19T01:23:28.498228+0000","last_undegraded":"2024-03-19T01:23:28.498228+0000","last_fullsized":"2024-03-19T01:23:28.498228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:42.729108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499902+0000","last_change":"2024-03-19T01:23:27.535733+0000","last_active":"2024-03-19T01:23:28.499902+0000","last_peered":"2024-03-19T01:23:28.499902+0000","last_clean":"2024-03-19T01:23:28.499902+0000","last_became_active":"2024-03-19T01:23:27.529249+0000","last_became_peered":"2024-03-19T01:23:27.529249+0000","last_unstale":"2024-03-19T01:23:28.499902+0000","last_undegraded":"2024-03-19T01:23:28.499902+0000","last_fullsized":"2024-03-19T01:23:28.499902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:33.934449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572947+0000","last_change":"2024-03-19T01:23:26.518355+0000","last_active":"2024-03-19T01:23:28.572947+0000","last_peered":"2024-03-19T01:23:28.572947+0000","last_clean":"2024-03-19T01:23:28.572947+0000","last_became_active":"2024-03-19T01:23:26.517357+0000","last_became_peered":"2024-03-19T01:23:26.517357+0000","last_unstale":"2024-03-19T01:23:28.572947+0000","last_undegraded":"2024-03-19T01:23:28.572947+0000","last_fullsized":"2024-03-19T01:23:28.572947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.565860+0000","last_change":"2024-03-19T01:23:27.564792+0000","last_active":"2024-03-19T01:23:27.565860+0000","last_peered":"2024-03-19T01:23:27.565860+0000","last_clean":"2024-03-19T01:23:27.565860+0000","last_became_active":"2024-03-19T01:23:27.564199+0000","last_became_peered":"2024-03-19T01:23:27.564199+0000","last_unstale":"2024-03-19T01:23:27.565860+0000","last_undegraded":"2024-03-19T01:23:27.565860+0000","last_fullsized":"2024-03-19T01:23:27.565860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.507342+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:23:27.507342+0000","last_peered":"2024-03-19T01:23:27.507342+0000","last_clean":"2024-03-19T01:23:27.507342+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:23:27.507342+0000","last_undegraded":"2024-03-19T01:23:27.507342+0000","last_fullsized":"2024-03-19T01:23:27.507342+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498418+0000","last_change":"2024-03-19T01:23:26.516922+0000","last_active":"2024-03-19T01:23:28.498418+0000","last_peered":"2024-03-19T01:23:28.498418+0000","last_clean":"2024-03-19T01:23:28.498418+0000","last_became_active":"2024-03-19T01:23:26.516206+0000","last_became_peered":"2024-03-19T01:23:26.516206+0000","last_unstale":"2024-03-19T01:23:28.498418+0000","last_undegraded":"2024-03-19T01:23:28.498418+0000","last_fullsized":"2024-03-19T01:23:28.498418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569215+0000","last_change":"2024-03-19T01:23:27.557742+0000","last_active":"2024-03-19T01:23:28.569215+0000","last_peered":"2024-03-19T01:23:28.569215+0000","last_clean":"2024-03-19T01:23:28.569215+0000","last_became_active":"2024-03-19T01:23:27.557487+0000","last_became_peered":"2024-03-19T01:23:27.557487+0000","last_unstale":"2024-03-19T01:23:28.569215+0000","last_undegraded":"2024-03-19T01:23:28.569215+0000","last_fullsized":"2024-03-19T01:23:28.569215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.507514+0000","last_change":"2024-03-19T01:23:26.539730+0000","last_active":"2024-03-19T01:23:27.507514+0000","last_peered":"2024-03-19T01:23:27.507514+0000","last_clean":"2024-03-19T01:23:27.507514+0000","last_became_active":"2024-03-19T01:23:26.538551+0000","last_became_peered":"2024-03-19T01:23:26.538551+0000","last_unstale":"2024-03-19T01:23:27.507514+0000","last_undegraded":"2024-03-19T01:23:27.507514+0000","last_fullsized":"2024-03-19T01:23:27.507514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.524654+0000","last_change":"2024-03-19T01:23:27.524267+0000","last_active":"2024-03-19T01:23:27.524654+0000","last_peered":"2024-03-19T01:23:27.524654+0000","last_clean":"2024-03-19T01:23:27.524654+0000","last_became_active":"2024-03-19T01:23:27.523862+0000","last_became_peered":"2024-03-19T01:23:27.523862+0000","last_unstale":"2024-03-19T01:23:27.524654+0000","last_undegraded":"2024-03-19T01:23:27.524654+0000","last_fullsized":"2024-03-19T01:23:27.524654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572805+0000","last_change":"2024-03-19T01:23:26.516027+0000","last_active":"2024-03-19T01:23:28.572805+0000","last_peered":"2024-03-19T01:23:28.572805+0000","last_clean":"2024-03-19T01:23:28.572805+0000","last_became_active":"2024-03-19T01:23:26.515582+0000","last_became_peered":"2024-03-19T01:23:26.515582+0000","last_unstale":"2024-03-19T01:23:28.572805+0000","last_undegraded":"2024-03-19T01:23:28.572805+0000","last_fullsized":"2024-03-19T01:23:28.572805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498092+0000","last_change":"2024-03-19T01:23:27.515653+0000","last_active":"2024-03-19T01:23:28.498092+0000","last_peered":"2024-03-19T01:23:28.498092+0000","last_clean":"2024-03-19T01:23:28.498092+0000","last_became_active":"2024-03-19T01:23:27.511685+0000","last_became_peered":"2024-03-19T01:23:27.511685+0000","last_unstale":"2024-03-19T01:23:28.498092+0000","last_undegraded":"2024-03-19T01:23:28.498092+0000","last_fullsized":"2024-03-19T01:23:28.498092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499532+0000","last_change":"2024-03-19T01:23:26.511797+0000","last_active":"2024-03-19T01:23:28.499532+0000","last_peered":"2024-03-19T01:23:28.499532+0000","last_clean":"2024-03-19T01:23:28.499532+0000","last_became_active":"2024-03-19T01:23:26.511112+0000","last_became_peered":"2024-03-19T01:23:26.511112+0000","last_unstale":"2024-03-19T01:23:28.499532+0000","last_undegraded":"2024-03-19T01:23:28.499532+0000","last_fullsized":"2024-03-19T01:23:28.499532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830892+0000","last_change":"2024-03-19T01:23:27.523591+0000","last_active":"2024-03-19T01:23:28.830892+0000","last_peered":"2024-03-19T01:23:28.830892+0000","last_clean":"2024-03-19T01:23:28.830892+0000","last_became_active":"2024-03-19T01:23:27.522418+0000","last_became_peered":"2024-03-19T01:23:27.522418+0000","last_unstale":"2024-03-19T01:23:28.830892+0000","last_undegraded":"2024-03-19T01:23:28.830892+0000","last_fullsized":"2024-03-19T01:23:28.830892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498246+0000","last_change":"2024-03-19T01:23:26.519145+0000","last_active":"2024-03-19T01:23:28.498246+0000","last_peered":"2024-03-19T01:23:28.498246+0000","last_clean":"2024-03-19T01:23:28.498246+0000","last_became_active":"2024-03-19T01:23:26.518686+0000","last_became_peered":"2024-03-19T01:23:26.518686+0000","last_unstale":"2024-03-19T01:23:28.498246+0000","last_undegraded":"2024-03-19T01:23:28.498246+0000","last_fullsized":"2024-03-19T01:23:28.498246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499719+0000","last_change":"2024-03-19T01:23:27.549327+0000","last_active":"2024-03-19T01:23:28.499719+0000","last_peered":"2024-03-19T01:23:28.499719+0000","last_clean":"2024-03-19T01:23:28.499719+0000","last_became_active":"2024-03-19T01:23:27.548727+0000","last_became_peered":"2024-03-19T01:23:27.548727+0000","last_unstale":"2024-03-19T01:23:28.499719+0000","last_undegraded":"2024-03-19T01:23:28.499719+0000","last_fullsized":"2024-03-19T01:23:28.499719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:56.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.524302+0000","last_change":"2024-03-19T01:23:26.539153+0000","last_active":"2024-03-19T01:23:27.524302+0000","last_peered":"2024-03-19T01:23:27.524302+0000","last_clean":"2024-03-19T01:23:27.524302+0000","last_became_active":"2024-03-19T01:23:26.537687+0000","last_became_peered":"2024-03-19T01:23:26.537687+0000","last_unstale":"2024-03-19T01:23:27.524302+0000","last_undegraded":"2024-03-19T01:23:27.524302+0000","last_fullsized":"2024-03-19T01:23:27.524302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:59.899930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497941+0000","last_change":"2024-03-19T01:23:27.508732+0000","last_active":"2024-03-19T01:23:28.497941+0000","last_peered":"2024-03-19T01:23:28.497941+0000","last_clean":"2024-03-19T01:23:28.497941+0000","last_became_active":"2024-03-19T01:23:27.508329+0000","last_became_peered":"2024-03-19T01:23:27.508329+0000","last_unstale":"2024-03-19T01:23:28.497941+0000","last_undegraded":"2024-03-19T01:23:28.497941+0000","last_fullsized":"2024-03-19T01:23:28.497941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498756+0000","last_change":"2024-03-19T01:23:26.536445+0000","last_active":"2024-03-19T01:23:28.498756+0000","last_peered":"2024-03-19T01:23:28.498756+0000","last_clean":"2024-03-19T01:23:28.498756+0000","last_became_active":"2024-03-19T01:23:26.535968+0000","last_became_peered":"2024-03-19T01:23:26.535968+0000","last_unstale":"2024-03-19T01:23:28.498756+0000","last_undegraded":"2024-03-19T01:23:28.498756+0000","last_fullsized":"2024-03-19T01:23:28.498756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569863+0000","last_change":"2024-03-19T01:23:27.553070+0000","last_active":"2024-03-19T01:23:28.569863+0000","last_peered":"2024-03-19T01:23:28.569863+0000","last_clean":"2024-03-19T01:23:28.569863+0000","last_became_active":"2024-03-19T01:23:27.552794+0000","last_became_peered":"2024-03-19T01:23:27.552794+0000","last_unstale":"2024-03-19T01:23:28.569863+0000","last_undegraded":"2024-03-19T01:23:28.569863+0000","last_fullsized":"2024-03-19T01:23:28.569863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.554475+0000","last_change":"2024-03-19T01:23:26.537527+0000","last_active":"2024-03-19T01:23:27.554475+0000","last_peered":"2024-03-19T01:23:27.554475+0000","last_clean":"2024-03-19T01:23:27.554475+0000","last_became_active":"2024-03-19T01:23:26.536523+0000","last_became_peered":"2024-03-19T01:23:26.536523+0000","last_unstale":"2024-03-19T01:23:27.554475+0000","last_undegraded":"2024-03-19T01:23:27.554475+0000","last_fullsized":"2024-03-19T01:23:27.554475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.514918+0000","last_change":"2024-03-19T01:23:27.511086+0000","last_active":"2024-03-19T01:23:27.514918+0000","last_peered":"2024-03-19T01:23:27.514918+0000","last_clean":"2024-03-19T01:23:27.514918+0000","last_became_active":"2024-03-19T01:23:27.510714+0000","last_became_peered":"2024-03-19T01:23:27.510714+0000","last_unstale":"2024-03-19T01:23:27.514918+0000","last_undegraded":"2024-03-19T01:23:27.514918+0000","last_fullsized":"2024-03-19T01:23:27.514918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:15.544112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.503246+0000","last_change":"2024-03-19T01:23:26.539154+0000","last_active":"2024-03-19T01:23:27.503246+0000","last_peered":"2024-03-19T01:23:27.503246+0000","last_clean":"2024-03-19T01:23:27.503246+0000","last_became_active":"2024-03-19T01:23:26.537837+0000","last_became_peered":"2024-03-19T01:23:26.537837+0000","last_unstale":"2024-03-19T01:23:27.503246+0000","last_undegraded":"2024-03-19T01:23:27.503246+0000","last_fullsized":"2024-03-19T01:23:27.503246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830757+0000","last_change":"2024-03-19T01:23:27.512029+0000","last_active":"2024-03-19T01:23:28.830757+0000","last_peered":"2024-03-19T01:23:28.830757+0000","last_clean":"2024-03-19T01:23:28.830757+0000","last_became_active":"2024-03-19T01:23:27.511155+0000","last_became_peered":"2024-03-19T01:23:27.511155+0000","last_unstale":"2024-03-19T01:23:28.830757+0000","last_undegraded":"2024-03-19T01:23:28.830757+0000","last_fullsized":"2024-03-19T01:23:28.830757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:38.911150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497954+0000","last_change":"2024-03-19T01:23:26.528143+0000","last_active":"2024-03-19T01:23:28.497954+0000","last_peered":"2024-03-19T01:23:28.497954+0000","last_clean":"2024-03-19T01:23:28.497954+0000","last_became_active":"2024-03-19T01:23:26.527487+0000","last_became_peered":"2024-03-19T01:23:26.527487+0000","last_unstale":"2024-03-19T01:23:28.497954+0000","last_undegraded":"2024-03-19T01:23:28.497954+0000","last_fullsized":"2024-03-19T01:23:28.497954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:07.823972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569510+0000","last_change":"2024-03-19T01:23:27.518443+0000","last_active":"2024-03-19T01:23:28.569510+0000","last_peered":"2024-03-19T01:23:28.569510+0000","last_clean":"2024-03-19T01:23:28.569510+0000","last_became_active":"2024-03-19T01:23:27.518170+0000","last_became_peered":"2024-03-19T01:23:27.518170+0000","last_unstale":"2024-03-19T01:23:28.569510+0000","last_undegraded":"2024-03-19T01:23:28.569510+0000","last_fullsized":"2024-03-19T01:23:28.569510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498375+0000","last_change":"2024-03-19T01:23:26.511877+0000","last_active":"2024-03-19T01:23:28.498375+0000","last_peered":"2024-03-19T01:23:28.498375+0000","last_clean":"2024-03-19T01:23:28.498375+0000","last_became_active":"2024-03-19T01:23:26.511439+0000","last_became_peered":"2024-03-19T01:23:26.511439+0000","last_unstale":"2024-03-19T01:23:28.498375+0000","last_undegraded":"2024-03-19T01:23:28.498375+0000","last_fullsized":"2024-03-19T01:23:28.498375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569314+0000","last_change":"2024-03-19T01:23:27.523071+0000","last_active":"2024-03-19T01:23:28.569314+0000","last_peered":"2024-03-19T01:23:28.569314+0000","last_clean":"2024-03-19T01:23:28.569314+0000","last_became_active":"2024-03-19T01:23:27.522721+0000","last_became_peered":"2024-03-19T01:23:27.522721+0000","last_unstale":"2024-03-19T01:23:28.569314+0000","last_undegraded":"2024-03-19T01:23:28.569314+0000","last_fullsized":"2024-03-19T01:23:28.569314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556257+0000","last_change":"2024-03-19T01:23:26.543568+0000","last_active":"2024-03-19T01:23:27.556257+0000","last_peered":"2024-03-19T01:23:27.556257+0000","last_clean":"2024-03-19T01:23:27.556257+0000","last_became_active":"2024-03-19T01:23:26.542859+0000","last_became_peered":"2024-03-19T01:23:26.542859+0000","last_unstale":"2024-03-19T01:23:27.556257+0000","last_undegraded":"2024-03-19T01:23:27.556257+0000","last_fullsized":"2024-03-19T01:23:27.556257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:42.729108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.551540+0000","last_change":"2024-03-19T01:23:27.547084+0000","last_active":"2024-03-19T01:23:27.551540+0000","last_peered":"2024-03-19T01:23:27.551540+0000","last_clean":"2024-03-19T01:23:27.551540+0000","last_became_active":"2024-03-19T01:23:27.532180+0000","last_became_peered":"2024-03-19T01:23:27.532180+0000","last_unstale":"2024-03-19T01:23:27.551540+0000","last_undegraded":"2024-03-19T01:23:27.551540+0000","last_fullsized":"2024-03-19T01:23:27.551540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499777+0000","last_change":"2024-03-19T01:23:27.557382+0000","last_active":"2024-03-19T01:23:28.499777+0000","last_peered":"2024-03-19T01:23:28.499777+0000","last_clean":"2024-03-19T01:23:28.499777+0000","last_became_active":"2024-03-19T01:23:27.556485+0000","last_became_peered":"2024-03-19T01:23:27.556485+0000","last_unstale":"2024-03-19T01:23:28.499777+0000","last_undegraded":"2024-03-19T01:23:28.499777+0000","last_fullsized":"2024-03-19T01:23:28.499777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:01.486458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.563879+0000","last_change":"2024-03-19T01:23:26.541210+0000","last_active":"2024-03-19T01:23:27.563879+0000","last_peered":"2024-03-19T01:23:27.563879+0000","last_clean":"2024-03-19T01:23:27.563879+0000","last_became_active":"2024-03-19T01:23:26.540535+0000","last_became_peered":"2024-03-19T01:23:26.540535+0000","last_unstale":"2024-03-19T01:23:27.563879+0000","last_undegraded":"2024-03-19T01:23:27.563879+0000","last_fullsized":"2024-03-19T01:23:27.563879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:07.823972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499107+0000","last_change":"2024-03-19T01:23:27.547119+0000","last_active":"2024-03-19T01:23:28.499107+0000","last_peered":"2024-03-19T01:23:28.499107+0000","last_clean":"2024-03-19T01:23:28.499107+0000","last_became_active":"2024-03-19T01:23:27.537831+0000","last_became_peered":"2024-03-19T01:23:27.537831+0000","last_unstale":"2024-03-19T01:23:28.499107+0000","last_undegraded":"2024-03-19T01:23:28.499107+0000","last_fullsized":"2024-03-19T01:23:28.499107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.551073+0000","last_change":"2024-03-19T01:23:26.544693+0000","last_active":"2024-03-19T01:23:27.551073+0000","last_peered":"2024-03-19T01:23:27.551073+0000","last_clean":"2024-03-19T01:23:27.551073+0000","last_became_active":"2024-03-19T01:23:26.544047+0000","last_became_peered":"2024-03-19T01:23:26.544047+0000","last_unstale":"2024-03-19T01:23:27.551073+0000","last_undegraded":"2024-03-19T01:23:27.551073+0000","last_fullsized":"2024-03-19T01:23:27.551073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:36.870147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.550564+0000","last_change":"2024-03-19T01:23:27.549737+0000","last_active":"2024-03-19T01:23:27.550564+0000","last_peered":"2024-03-19T01:23:27.550564+0000","last_clean":"2024-03-19T01:23:27.550564+0000","last_became_active":"2024-03-19T01:23:27.549460+0000","last_became_peered":"2024-03-19T01:23:27.549460+0000","last_unstale":"2024-03-19T01:23:27.550564+0000","last_undegraded":"2024-03-19T01:23:27.550564+0000","last_fullsized":"2024-03-19T01:23:27.550564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:00.911501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569507+0000","last_change":"2024-03-19T01:23:26.520272+0000","last_active":"2024-03-19T01:23:28.569507+0000","last_peered":"2024-03-19T01:23:28.569507+0000","last_clean":"2024-03-19T01:23:28.569507+0000","last_became_active":"2024-03-19T01:23:26.519940+0000","last_became_peered":"2024-03-19T01:23:26.519940+0000","last_unstale":"2024-03-19T01:23:28.569507+0000","last_undegraded":"2024-03-19T01:23:28.569507+0000","last_fullsized":"2024-03-19T01:23:28.569507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:10.187739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569312+0000","last_change":"2024-03-19T01:23:27.541353+0000","last_active":"2024-03-19T01:23:28.569312+0000","last_peered":"2024-03-19T01:23:28.569312+0000","last_clean":"2024-03-19T01:23:28.569312+0000","last_became_active":"2024-03-19T01:23:27.541054+0000","last_became_peered":"2024-03-19T01:23:27.541054+0000","last_unstale":"2024-03-19T01:23:28.569312+0000","last_undegraded":"2024-03-19T01:23:28.569312+0000","last_fullsized":"2024-03-19T01:23:28.569312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-19T01:23:27.556949+0000","last_change":"2024-03-19T01:23:26.528906+0000","last_active":"2024-03-19T01:23:27.556949+0000","last_peered":"2024-03-19T01:23:27.556949+0000","last_clean":"2024-03-19T01:23:27.556949+0000","last_became_active":"2024-03-19T01:23:26.528592+0000","last_became_peered":"2024-03-19T01:23:26.528592+0000","last_unstale":"2024-03-19T01:23:27.556949+0000","last_undegraded":"2024-03-19T01:23:27.556949+0000","last_fullsized":"2024-03-19T01:23:27.556949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":660255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":69431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":212992,"data_stored":69465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":660311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:23:32.137 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-19T01:23:32.137 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-19T01:23:32.138 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-19T01:23:32.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-19T01:23:32.138 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-19T01:23:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.324+0000 7f44fcb0e640 1 Processor -- start 2024-03-19T01:23:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.324+0000 7f44fcb0e640 1 -- start start 2024-03-19T01:23:32.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44fcb0e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f44f80b1460 0x7f44f80b6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44fcb0e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f80b1840 con 0x7f44f8144e10 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44fcb0e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f80b2120 con 0x7f44f806d870 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44fcb0e640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f44f81441e0 con 0x7f44f80b1460 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f6d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f44f8144e10 0x7f44f814e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53150/0 (socket says 172.21.15.7:53150) 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f6d9d640 1 -- 172.21.15.7:0/1429395258 learned_addr learned my addr 172.21.15.7:0/1429395258 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f659c640 1 --2- 172.21.15.7:0/1429395258 >> v2:172.21.15.167:3300/0 conn(0x7f44f80b1460 0x7f44f80b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866256451 0 0) 0x7f44f80b1840 con 0x7f44f8144e10 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e4003530 con 0x7f44f8144e10 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 667385694 0 0) 0x7f44e4003530 con 0x7f44f8144e10 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 >> v2:172.21.15.167:3300/0 conn(0x7f44f80b1460 msgr2=0x7f44f80b6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 --2- 172.21.15.7:0/1429395258 >> v2:172.21.15.167:3300/0 conn(0x7f44f80b1460 0x7f44f80b6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 >> v1:172.21.15.167:6789/0 conn(0x7f44f806d870 legacy=0x7f44f8144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f80b85c0 con 0x7f44f8144e10 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f44f0001680 con 0x7f44f8144e10 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 >> v1:172.21.15.7:6789/0 conn(0x7f44f8144e10 legacy=0x7f44f814e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44e40039c0 0x7f44e4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44e4004810 0x7f44e4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44e400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f44f81441e0 con 0x7f44e40051d0 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f44e400c170 con 0x7f44e40039c0 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f44e400e580 con 0x7f44e4004810 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f6d9d640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44e400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f659c640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44e40039c0 0x7f44e4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f5d9b640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44e4004810 0x7f44e4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f6d9d640 1 -- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44e4004810 msgr2=0x7f44e4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f6d9d640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44e4004810 0x7f44e4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f6d9d640 1 -- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44e40039c0 msgr2=0x7f44e4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f6d9d640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44e40039c0 0x7f44e4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f6d9d640 1 -- 172.21.15.7:0/1429395258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e4003530 con 0x7f44e40051d0 2024-03-19T01:23:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f659c640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44e40039c0 0x7f44e4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.328+0000 7f44f5d9b640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44e4004810 0x7f44e4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f6d9d640 1 -- 172.21.15.7:0/1429395258 reap_dead start 2024-03-19T01:23:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f6d9d640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44e400b9d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f44f80b6f40 tx=0x7f44f001f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc09c4f9dad4312 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0007d40 con 0x7f44e40051d0 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f0005ae0 con 0x7f44e40051d0 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f559a640 1 -- 172.21.15.7:0/1429395258 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0007070 con 0x7f44e40051d0 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 -- 172.21.15.7:0/1429395258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 msgr2=0x7f44e400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44e400b9d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f44f80b6f40 tx=0x7f44f001f370 comp rx=0 tx=0).stop 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 -- 172.21.15.7:0/1429395258 shutdown_connections 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 --2- 172.21.15.7:0/1429395258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44e400b9d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 -- 172.21.15.7:0/1429395258 >> 172.21.15.7:0/1429395258 conn(0x7f44f80b5330 msgr2=0x7f44f80b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 -- 172.21.15.7:0/1429395258 shutdown_connections 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 -- 172.21.15.7:0/1429395258 wait complete. 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 Processor -- start 2024-03-19T01:23:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 -- start start 2024-03-19T01:23:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44f8138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f8071890 0x7f44f81dc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f81369d0 0x7f44f8136e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f44f8144880 con 0x7f44e40051d0 2024-03-19T01:23:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f44f8144700 con 0x7f44f81369d0 2024-03-19T01:23:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44fcb0e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f44f8144a00 con 0x7f44f8071890 2024-03-19T01:23:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44f8138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44f8138190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43176/0 (socket says 172.21.15.7:43176) 2024-03-19T01:23:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f659c640 1 -- 172.21.15.7:0/3623054103 learned_addr learned my addr 172.21.15.7:0/3623054103 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f5d9b640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f8071890 0x7f44f81dc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f6d9d640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f81369d0 0x7f44f8136e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f659c640 1 -- 172.21.15.7:0/3623054103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f8071890 msgr2=0x7f44f81dc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f659c640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f8071890 0x7f44f81dc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f659c640 1 -- 172.21.15.7:0/3623054103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f81369d0 msgr2=0x7f44f8136e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f659c640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f81369d0 0x7f44f8136e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f659c640 1 -- 172.21.15.7:0/3623054103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f81cf160 con 0x7f44e40051d0 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.332+0000 7f44f6d9d640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f81369d0 0x7f44f8136e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44f659c640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44f8138190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009fd0 tx=0x7f44e0011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44eeffd640 1 -- 172.21.15.7:0/3623054103 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0004220 con 0x7f44e40051d0 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44eeffd640 1 -- 172.21.15.7:0/3623054103 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e00043c0 con 0x7f44e40051d0 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f81394d0 con 0x7f44e40051d0 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44eeffd640 1 -- 172.21.15.7:0/3623054103 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0015890 con 0x7f44e40051d0 2024-03-19T01:23:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f81d01e0 con 0x7f44e40051d0 2024-03-19T01:23:32.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f8075460 con 0x7f44e40051d0 2024-03-19T01:23:32.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44eeffd640 1 -- 172.21.15.7:0/3623054103 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f44e0015a30 con 0x7f44e40051d0 2024-03-19T01:23:32.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44eeffd640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d8067660 0x7f44d8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44eeffd640 1 -- 172.21.15.7:0/3623054103 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f44e0085df0 con 0x7f44e40051d0 2024-03-19T01:23:32.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44f5d9b640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d8067660 0x7f44d8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.336+0000 7f44f5d9b640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d8067660 0x7f44d8069b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f44f81dc180 tx=0x7f44e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:32.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.340+0000 7f44eeffd640 1 -- 172.21.15.7:0/3623054103 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f44e0019030 con 0x7f44e40051d0 2024-03-19T01:23:32.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.572+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f44f81431a0 con 0x7f44e40051d0 2024-03-19T01:23:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44eeffd640 1 -- 172.21.15.7:0/3623054103 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+166518 (secure 0 0 0) 0x7f44e0056b00 con 0x7f44e40051d0 2024-03-19T01:23:32.584 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:32.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d8067660 msgr2=0x7f44d8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d8067660 0x7f44d8069b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f44f81dc180 tx=0x7f44e8009040 comp rx=0 tx=0).stop 2024-03-19T01:23:32.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 msgr2=0x7f44f8138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44f8138190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009fd0 tx=0x7f44e0011a60 comp rx=0 tx=0).stop 2024-03-19T01:23:32.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 shutdown_connections 2024-03-19T01:23:32.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44d8067660 0x7f44d8069b20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44f8071890 0x7f44f81dc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44f81369d0 0x7f44f8136e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 --2- 172.21.15.7:0/3623054103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e40051d0 0x7f44f8138190 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 >> 172.21.15.7:0/3623054103 conn(0x7f44f80b5330 msgr2=0x7f44f80b8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:32.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 shutdown_connections 2024-03-19T01:23:32.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.580+0000 7f44fcb0e640 1 -- 172.21.15.7:0/3623054103 wait complete. 2024-03-19T01:23:32.603 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1075636442},{"type":"v1","addr":"172.21.15.7:6833","nonce":1075636442}]},"active_addr":"172.21.15.7:6833/1075636442","active_change":"2024-03-19T01:23:06.874860+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4117,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":34875796}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":4093539693}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1821724249}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3475302635}]}]} 2024-03-19T01:23:32.606 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-19T01:23:32.606 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-19T01:23:32.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:32.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6843cd9640 1 Processor -- start 2024-03-19T01:23:32.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6843cd9640 1 -- start start 2024-03-19T01:23:32.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6843cd9640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f683c0701a0 0x7f683c13f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6843cd9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c145e20 0x7f683c0b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6843cd9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f683c0711c0 con 0x7f683c074390 2024-03-19T01:23:32.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6843cd9640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f683c071430 con 0x7f683c0701a0 2024-03-19T01:23:32.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6843cd9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f683c074770 con 0x7f683c145e20 2024-03-19T01:23:32.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841a75640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f683c074390 0x7f683c13eec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43130/0 (socket says 172.21.15.7:43130) 2024-03-19T01:23:32.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841a75640 1 -- 172.21.15.7:0/3264864307 learned_addr learned my addr 172.21.15.7:0/3264864307 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:32.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841274640 1 --2- 172.21.15.7:0/3264864307 >> v2:172.21.15.167:3300/0 conn(0x7f683c0701a0 0x7f683c13f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6842276640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c145e20 0x7f683c0b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705709967 0 0) 0x7f683c0711c0 con 0x7f683c074390 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6828003530 con 0x7f683c074390 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3426699070 0 0) 0x7f6828003530 con 0x7f683c074390 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c145e20 msgr2=0x7f683c0b6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c145e20 0x7f683c0b6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 >> v2:172.21.15.167:3300/0 conn(0x7f683c0701a0 msgr2=0x7f683c13f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 --2- 172.21.15.7:0/3264864307 >> v2:172.21.15.167:3300/0 conn(0x7f683c0701a0 0x7f683c13f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683c145340 con 0x7f683c074390 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f6838001680 con 0x7f683c074390 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 >> v1:172.21.15.7:6789/0 conn(0x7f683c074390 legacy=0x7f683c13eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68280039c0 0x7f6828003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c145e20 0x7f6828004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6828004d50 0x7f68280051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f683c071430 con 0x7f68280039c0 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f683c074770 con 0x7f6828004d50 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f682800b9d0 con 0x7f683c145e20 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841a75640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68280039c0 0x7f6828003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6842276640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6828004d50 0x7f68280051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841274640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c145e20 0x7f6828004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841a75640 1 -- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c145e20 msgr2=0x7f6828004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841a75640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c145e20 0x7f6828004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841a75640 1 -- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6828004d50 msgr2=0x7f68280051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841a75640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6828004d50 0x7f68280051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.720+0000 7f6841a75640 1 -- 172.21.15.7:0/3264864307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6828003530 con 0x7f68280039c0 2024-03-19T01:23:32.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6841a75640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68280039c0 0x7f6828003de0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f683800c240 tx=0x7f683801cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d453a7462eef00a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838007d40 con 0x7f68280039c0 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68380a9020 con 0x7f68280039c0 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6840a73640 1 -- 172.21.15.7:0/3264864307 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838005b00 con 0x7f68280039c0 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 -- 172.21.15.7:0/3264864307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68280039c0 msgr2=0x7f6828003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 --2- 172.21.15.7:0/3264864307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68280039c0 0x7f6828003de0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f683800c240 tx=0x7f683801cc70 comp rx=0 tx=0).stop 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6842276640 1 -- 172.21.15.7:0/3264864307 reap_dead start 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 -- 172.21.15.7:0/3264864307 shutdown_connections 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 -- 172.21.15.7:0/3264864307 >> 172.21.15.7:0/3264864307 conn(0x7f683c0b9170 msgr2=0x7f683c0b9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 -- 172.21.15.7:0/3264864307 shutdown_connections 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 -- 172.21.15.7:0/3264864307 wait complete. 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 Processor -- start 2024-03-19T01:23:32.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 -- start start 2024-03-19T01:23:32.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68280039c0 0x7f683c13a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c1dfbc0 0x7f683c1e0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f683c1384f0 0x7f683c138970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f683c06fa00 con 0x7f683c1384f0 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f683c0b7340 con 0x7f68280039c0 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6843cd9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f683c1503c0 con 0x7f683c1dfbc0 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6842276640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f683c1384f0 0x7f683c138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.724+0000 7f6842276640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f683c1384f0 0x7f683c138970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39796/0 (socket says 172.21.15.7:39796) 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6842276640 1 -- 172.21.15.7:0/696990778 learned_addr learned my addr 172.21.15.7:0/696990778 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6841a75640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68280039c0 0x7f683c13a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6841274640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c1dfbc0 0x7f683c1e0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6842276640 1 -- 172.21.15.7:0/696990778 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c1dfbc0 msgr2=0x7f683c1e0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6842276640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c1dfbc0 0x7f683c1e0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6842276640 1 -- 172.21.15.7:0/696990778 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68280039c0 msgr2=0x7f683c13a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6842276640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68280039c0 0x7f683c13a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6842276640 1 -- 172.21.15.7:0/696990778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683c1e0580 con 0x7f683c1384f0 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6842276640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f683c1384f0 0x7f683c138970 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f683c0b37b0 tx=0x7f68340115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:32.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f682e7fc640 1 -- 172.21.15.7:0/696990778 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834011de0 con 0x7f683c1384f0 2024-03-19T01:23:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f682e7fc640 1 -- 172.21.15.7:0/696990778 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f683401b070 con 0x7f683c1384f0 2024-03-19T01:23:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f682e7fc640 1 -- 172.21.15.7:0/696990778 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68340039a0 con 0x7f683c1384f0 2024-03-19T01:23:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f683c13acc0 con 0x7f683c1384f0 2024-03-19T01:23:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f683c142730 con 0x7f683c1384f0 2024-03-19T01:23:32.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f683c074a70 con 0x7f683c1384f0 2024-03-19T01:23:32.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f682e7fc640 1 -- 172.21.15.7:0/696990778 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6834003b40 con 0x7f683c1384f0 2024-03-19T01:23:32.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f682e7fc640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6820067660 0x7f6820069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:32.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6841a75640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6820067660 0x7f6820069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:32.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f682e7fc640 1 -- 172.21.15.7:0/696990778 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f68340860b0 con 0x7f683c1384f0 2024-03-19T01:23:32.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.728+0000 7f6841a75640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6820067660 0x7f6820069b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f68380aa000 tx=0x7f683c074920 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:32.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.732+0000 7f682e7fc640 1 -- 172.21.15.7:0/696990778 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6834025020 con 0x7f683c1384f0 2024-03-19T01:23:32.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f683c0adf60 con 0x7f683c1384f0 2024-03-19T01:23:32.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f682e7fc640 1 -- 172.21.15.7:0/696990778 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7f683c1e0580 con 0x7f683c1384f0 2024-03-19T01:23:32.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:32.901 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6820067660 msgr2=0x7f6820069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6820067660 0x7f6820069b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f68380aa000 tx=0x7f683c074920 comp rx=0 tx=0).stop 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f683c1384f0 msgr2=0x7f683c138970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f683c1384f0 0x7f683c138970 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f683c0b37b0 tx=0x7f68340115a0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 shutdown_connections 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6820067660 0x7f6820069b20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f683c1dfbc0 0x7f683c1e0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f683c1384f0 0x7f683c138970 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 --2- 172.21.15.7:0/696990778 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68280039c0 0x7f683c13a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 >> 172.21.15.7:0/696990778 conn(0x7f683c0b9170 msgr2=0x7f683c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.896+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 shutdown_connections 2024-03-19T01:23:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:32.900+0000 7f6843cd9640 1 -- 172.21.15.7:0/696990778 wait complete. 2024-03-19T01:23:32.920 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-19T01:23:32.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:33.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc8401c0640 1 Processor -- start 2024-03-19T01:23:33.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc8401c0640 1 -- start start 2024-03-19T01:23:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc8401c0640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fc83813df30 0x7fc83813e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc8401c0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838144340 0x7fc8381447b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc8401c0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc838144cf0 con 0x7fc838074390 2024-03-19T01:23:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc8401c0640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fc83813e900 con 0x7fc83813df30 2024-03-19T01:23:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc8401c0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc838074770 con 0x7fc838144340 2024-03-19T01:23:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83df5c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc838074390 0x7fc838147710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43136/0 (socket says 172.21.15.7:43136) 2024-03-19T01:23:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83df5c640 1 -- 172.21.15.7:0/3038240858 learned_addr learned my addr 172.21.15.7:0/3038240858 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83e75d640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838144340 0x7fc8381447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3452447778 0 0) 0x7fc838144cf0 con 0x7fc838074390 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83d75b640 1 --2- 172.21.15.7:0/3038240858 >> v2:172.21.15.167:3300/0 conn(0x7fc83813df30 0x7fc83813e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc820003530 con 0x7fc838074390 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1564534723 0 0) 0x7fc820003530 con 0x7fc838074390 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838144340 msgr2=0x7fc8381447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838144340 0x7fc8381447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 >> v2:172.21.15.167:3300/0 conn(0x7fc83813df30 msgr2=0x7fc83813e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 --2- 172.21.15.7:0/3038240858 >> v2:172.21.15.167:3300/0 conn(0x7fc83813df30 0x7fc83813e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc838147ec0 con 0x7fc838074390 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fc828001b80 con 0x7fc838074390 2024-03-19T01:23:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 >> v1:172.21.15.7:6789/0 conn(0x7fc838074390 legacy=0x7fc838147710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200039c0 0x7fc820003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc820004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838144340 0x7fc8200051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc838074770 con 0x7fc820004810 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc82000ba00 con 0x7fc8200039c0 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc82000bc00 con 0x7fc838144340 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83d75b640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc820004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83df5c640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200039c0 0x7fc820003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83e75d640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838144340 0x7fc8200051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83d75b640 1 -- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838144340 msgr2=0x7fc8200051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83d75b640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838144340 0x7fc8200051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83d75b640 1 -- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200039c0 msgr2=0x7fc820003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83d75b640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200039c0 0x7fc820003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.048+0000 7fc83d75b640 1 -- 172.21.15.7:0/3038240858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc820003530 con 0x7fc820004810 2024-03-19T01:23:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc83d75b640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc820004c90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009ae0 tx=0x7fc82c01cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1122ef45548a0dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c0096b0 con 0x7fc820004810 2024-03-19T01:23:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc82c0a6490 con 0x7fc820004810 2024-03-19T01:23:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc83cf5a640 1 -- 172.21.15.7:0/3038240858 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c0a6880 con 0x7fc820004810 2024-03-19T01:23:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 -- 172.21.15.7:0/3038240858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 msgr2=0x7fc820004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 --2- 172.21.15.7:0/3038240858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc820004c90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009ae0 tx=0x7fc82c01cab0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc83e75d640 1 -- 172.21.15.7:0/3038240858 reap_dead start 2024-03-19T01:23:33.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 -- 172.21.15.7:0/3038240858 shutdown_connections 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 -- 172.21.15.7:0/3038240858 >> 172.21.15.7:0/3038240858 conn(0x7fc8380b5c20 msgr2=0x7fc8380b6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 -- 172.21.15.7:0/3038240858 shutdown_connections 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 -- 172.21.15.7:0/3038240858 wait complete. 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 Processor -- start 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 -- start start 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc83813cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc838147270 0x7fc8381452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838143850 0x7fc838138540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc8380b3620 con 0x7fc820004810 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc8380b34a0 con 0x7fc838147270 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.052+0000 7fc8401c0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc83814f2b0 con 0x7fc838143850 2024-03-19T01:23:33.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83df5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc83813cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83df5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc83813cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39816/0 (socket says 172.21.15.7:39816) 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83df5c640 1 -- 172.21.15.7:0/2466144956 learned_addr learned my addr 172.21.15.7:0/2466144956 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83e75d640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838143850 0x7fc838138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83d75b640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc838147270 0x7fc8381452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83df5c640 1 -- 172.21.15.7:0/2466144956 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838143850 msgr2=0x7fc838138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83df5c640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838143850 0x7fc838138540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83df5c640 1 -- 172.21.15.7:0/2466144956 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc838147270 msgr2=0x7fc8381452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83df5c640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc838147270 0x7fc8381452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83df5c640 1 -- 172.21.15.7:0/2466144956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8380b00e0 con 0x7fc820004810 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83df5c640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc83813cbf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc828002240 tx=0x7fc82800ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc8367fc640 1 -- 172.21.15.7:0/2466144956 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc828005d40 con 0x7fc820004810 2024-03-19T01:23:33.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc8367fc640 1 -- 172.21.15.7:0/2466144956 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc828018070 con 0x7fc820004810 2024-03-19T01:23:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc8367fc640 1 -- 172.21.15.7:0/2466144956 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82800c9f0 con 0x7fc820004810 2024-03-19T01:23:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83813c6d0 con 0x7fc820004810 2024-03-19T01:23:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83813b970 con 0x7fc820004810 2024-03-19T01:23:33.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc83813c4d0 con 0x7fc820004810 2024-03-19T01:23:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc8367fc640 1 -- 172.21.15.7:0/2466144956 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc82800cb90 con 0x7fc820004810 2024-03-19T01:23:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc8367fc640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc824067690 0x7fc824069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc83d75b640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc824067690 0x7fc824069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.056+0000 7fc8367fc640 1 -- 172.21.15.7:0/2466144956 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc828085a00 con 0x7fc820004810 2024-03-19T01:23:33.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.060+0000 7fc83d75b640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc824067690 0x7fc824069b50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0ad000 tx=0x7fc82c0b2040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.060+0000 7fc8367fc640 1 -- 172.21.15.7:0/2466144956 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc828056710 con 0x7fc820004810 2024-03-19T01:23:33.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.224+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc83813c8a0 con 0x7fc820004810 2024-03-19T01:23:33.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8367fc640 1 -- 172.21.15.7:0/2466144956 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7fc828056010 con 0x7fc820004810 2024-03-19T01:23:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:33.231 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc824067690 msgr2=0x7fc824069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc824067690 0x7fc824069b50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0ad000 tx=0x7fc82c0b2040 comp rx=0 tx=0).stop 2024-03-19T01:23:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 msgr2=0x7fc83813cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc83813cbf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc828002240 tx=0x7fc82800ef80 comp rx=0 tx=0).stop 2024-03-19T01:23:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 shutdown_connections 2024-03-19T01:23:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc824067690 0x7fc824069b50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc838143850 0x7fc838138540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc838147270 0x7fc8381452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 --2- 172.21.15.7:0/2466144956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc820004810 0x7fc83813cbf0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.228+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 >> 172.21.15.7:0/2466144956 conn(0x7fc8380b5c20 msgr2=0x7fc83813adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.232+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 shutdown_connections 2024-03-19T01:23:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.232+0000 7fc8401c0640 1 -- 172.21.15.7:0/2466144956 wait complete. 2024-03-19T01:23:33.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-19T01:23:33.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-19T01:23:33.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-19T01:23:33.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-19T01:23:33.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-19T01:23:33.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-19T01:23:33.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-19T01:23:33.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-19T01:23:33.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e300b640 1 Processor -- start 2024-03-19T01:23:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f43fff640 1 Processor -- start 2024-03-19T01:23:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e300b640 1 -- start start 2024-03-19T01:23:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f43fff640 1 -- start start 2024-03-19T01:23:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e300b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29dc06b520 0x7f29dc0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e300b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc145d20 0x7f29dc072790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e300b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29dc06b900 con 0x7f29dc070240 2024-03-19T01:23:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e300b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f29dc071d60 con 0x7f29dc06b520 2024-03-19T01:23:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e300b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f29dc071f90 con 0x7f29dc145d20 2024-03-19T01:23:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f43fff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0f440721a0 0x7f0f44072630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f43fff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 0x7f0f441562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f43fff640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f440b1e90 con 0x7f0f4406c360 2024-03-19T01:23:33.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f43fff640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f0f4406c740 con 0x7f0f440721a0 2024-03-19T01:23:33.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f43fff640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0f440b2470 con 0x7f0f44153f00 2024-03-19T01:23:33.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e0da7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29dc06b520 0x7f29dc0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e0da7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29dc06b520 0x7f29dc0b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39828/0 (socket says 172.21.15.7:39828) 2024-03-19T01:23:33.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e0da7640 1 -- 172.21.15.7:0/3631616791 learned_addr learned my addr 172.21.15.7:0/3631616791 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e15a8640 1 --2- 172.21.15.7:0/3631616791 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc145d20 0x7f29dc072790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f42ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0f4406c360 0x7f0f440b06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43150/0 (socket says 172.21.15.7:43150) 2024-03-19T01:23:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f3bfff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0f440721a0 0x7f0f44072630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f437fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 0x7f0f441562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f3bfff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0f440721a0 0x7f0f44072630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45330/0 (socket says 172.21.15.7:45330) 2024-03-19T01:23:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f3bfff640 1 -- 172.21.15.7:0/1959267654 learned_addr learned my addr 172.21.15.7:0/1959267654 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f3bfff640 1 -- 172.21.15.7:0/1959267654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 msgr2=0x7f0f441562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f3bfff640 1 --2- 172.21.15.7:0/1959267654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 0x7f0f441562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f3bfff640 1 -- 172.21.15.7:0/1959267654 >> v1:172.21.15.7:6789/0 conn(0x7f0f4406c360 legacy=0x7f0f440b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f0f3bfff640 1 -- 172.21.15.7:0/1959267654 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f44072b70 con 0x7f0f440721a0 2024-03-19T01:23:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e0da7640 1 -- 172.21.15.7:0/3631616791 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc145d20 msgr2=0x7f29dc072790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e0da7640 1 --2- 172.21.15.7:0/3631616791 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc145d20 0x7f29dc072790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e0da7640 1 -- 172.21.15.7:0/3631616791 >> v1:172.21.15.167:6789/0 conn(0x7f29dc070240 legacy=0x7f29dc145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.444+0000 7f29e0da7640 1 -- 172.21.15.7:0/3631616791 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29dc072270 con 0x7f29dc06b520 2024-03-19T01:23:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.448+0000 7f29e0da7640 1 --2- 172.21.15.7:0/3631616791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 0x7f29dc0b4b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f29c8009870 tx=0x7f29c801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f1c022433595cc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.448+0000 7f29db7fe640 1 -- 172.21.15.7:0/3631616791 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c80ab5f0 con 0x7f29dc06b520 2024-03-19T01:23:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.448+0000 7f0f3bfff640 1 --2- 172.21.15.7:0/1959267654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f440721a0 0x7f0f44072630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009870 tx=0x7f0f2c01ce50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d25137c7db08e009 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.448+0000 7f29db7fe640 1 -- 172.21.15.7:0/3631616791 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c80b0070 con 0x7f29dc06b520 2024-03-19T01:23:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.448+0000 7f0f427fc640 1 -- 172.21.15.7:0/1959267654 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c0b0070 con 0x7f0f440721a0 2024-03-19T01:23:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.448+0000 7f0f427fc640 1 -- 172.21.15.7:0/1959267654 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f2c004230 con 0x7f0f440721a0 2024-03-19T01:23:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f0f43fff640 1 -- 172.21.15.7:0/1959267654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f440721a0 msgr2=0x7f0f44072630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f0f43fff640 1 --2- 172.21.15.7:0/1959267654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f440721a0 0x7f0f44072630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009870 tx=0x7f0f2c01ce50 comp rx=0 tx=0).stop 2024-03-19T01:23:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 -- 172.21.15.7:0/3631616791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 msgr2=0x7f29dc0b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 --2- 172.21.15.7:0/3631616791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 0x7f29dc0b4b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f29c8009870 tx=0x7f29c801c520 comp rx=0 tx=0).stop 2024-03-19T01:23:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 -- 172.21.15.7:0/3631616791 shutdown_connections 2024-03-19T01:23:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 --2- 172.21.15.7:0/3631616791 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc145d20 0x7f29dc072790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 --2- 172.21.15.7:0/3631616791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 0x7f29dc0b4b70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 -- 172.21.15.7:0/3631616791 >> 172.21.15.7:0/3631616791 conn(0x7f29dc074130 msgr2=0x7f29dc074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 -- 172.21.15.7:0/3631616791 shutdown_connections 2024-03-19T01:23:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f0f43fff640 1 -- 172.21.15.7:0/1959267654 shutdown_connections 2024-03-19T01:23:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f0f43fff640 1 --2- 172.21.15.7:0/1959267654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 0x7f0f441562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f0f43fff640 1 --2- 172.21.15.7:0/1959267654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f440721a0 0x7f0f44072630 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f0f43fff640 1 -- 172.21.15.7:0/1959267654 >> 172.21.15.7:0/1959267654 conn(0x7f0f440b8c60 msgr2=0x7f0f440b9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 -- 172.21.15.7:0/3631616791 wait complete. 2024-03-19T01:23:33.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 Processor -- start 2024-03-19T01:23:33.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f0f43fff640 1 -- 172.21.15.7:0/1959267654 shutdown_connections 2024-03-19T01:23:33.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f0f43fff640 1 -- 172.21.15.7:0/1959267654 wait complete. 2024-03-19T01:23:33.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 -- start start 2024-03-19T01:23:33.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 0x7f29dc0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc070240 0x7f29dc0af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc0afbb0 0x7f29dc1ec9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc158bd0 con 0x7f29dc06b520 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc158d50 con 0x7f29dc070240 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e300b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f29dc158ed0 con 0x7f29dc0afbb0 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e0da7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 0x7f29dc0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e0da7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 0x7f29dc0b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39844/0 (socket says 172.21.15.7:39844) 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e0da7640 1 -- 172.21.15.7:0/3183462873 learned_addr learned my addr 172.21.15.7:0/3183462873 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29e15a8640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc0afbb0 0x7f29dc1ec9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f29dbfff640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc070240 0x7f29dc0af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f0f43fff640 1 Processor -- start 2024-03-19T01:23:33.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f906a578640 1 Processor -- start 2024-03-19T01:23:33.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f906a578640 1 -- start start 2024-03-19T01:23:33.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f906a578640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c037c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f906a578640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f905c03c080 0x7f905c036a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f906a578640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f905c038570 con 0x7f905c03ba60 2024-03-19T01:23:33.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f906a578640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f905c038850 con 0x7f905c03c080 2024-03-19T01:23:33.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f906a578640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f905c038270 con 0x7f905c03b680 2024-03-19T01:23:33.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f9069d77640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f905c03ba60 0x7f905c0d0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:40728/0 (socket says 172.21.15.7:40728) 2024-03-19T01:23:33.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f9069d77640 1 -- 172.21.15.7:0/3967025396 learned_addr learned my addr 172.21.15.7:0/3967025396 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:33.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.452+0000 7f9068d75640 1 --2- 172.21.15.7:0/3967025396 >> v2:172.21.15.167:3300/0 conn(0x7f905c03c080 0x7f905c036a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 -- start start 2024-03-19T01:23:33.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29e0da7640 1 -- 172.21.15.7:0/3183462873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc0afbb0 msgr2=0x7f29dc1ec9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29e0da7640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc0afbb0 0x7f29dc1ec9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29e0da7640 1 -- 172.21.15.7:0/3183462873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc070240 msgr2=0x7f29dc0af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29e0da7640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc070240 0x7f29dc0af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29e0da7640 1 -- 172.21.15.7:0/3183462873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29dc0b3620 con 0x7f29dc06b520 2024-03-19T01:23:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff401712640 1 Processor -- start 2024-03-19T01:23:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff401712640 1 -- start start 2024-03-19T01:23:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f905bfff640 1 -- 172.21.15.7:0/3967025396 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991987428 0 0) 0x7f905c038570 con 0x7f905c03ba60 2024-03-19T01:23:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f905bfff640 1 -- 172.21.15.7:0/3967025396 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9050003530 con 0x7f905c03ba60 2024-03-19T01:23:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29e0da7640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 0x7f29dc0b2f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f29c801abd0 tx=0x7f29c801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f9069576640 1 --2- 172.21.15.7:0/3967025396 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c037c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29d97fa640 1 -- 172.21.15.7:0/3183462873 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c80b4800 con 0x7f29dc06b520 2024-03-19T01:23:33.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7fb41f59e640 1 Processor -- start 2024-03-19T01:23:33.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f4406c360 0x7f0f440ab270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f440721a0 0x7f0f441e48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 0x7f0f441e7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0f440b4a70 con 0x7f0f440721a0 2024-03-19T01:23:33.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0f440b48f0 con 0x7f0f4406c360 2024-03-19T01:23:33.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0f440b4c50 con 0x7f0f44153f00 2024-03-19T01:23:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f9069576640 1 -- 172.21.15.7:0/3967025396 >> v2:172.21.15.167:3300/0 conn(0x7f905c03c080 msgr2=0x7f905c036a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f9069576640 1 --2- 172.21.15.7:0/3967025396 >> v2:172.21.15.167:3300/0 conn(0x7f905c03c080 0x7f905c036a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f9069576640 1 -- 172.21.15.7:0/3967025396 >> v1:172.21.15.167:6789/0 conn(0x7f905c03ba60 legacy=0x7f905c0d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f9069576640 1 -- 172.21.15.7:0/3967025396 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c036f50 con 0x7f905c03b680 2024-03-19T01:23:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f905bfff640 1 -- 172.21.15.7:0/3967025396 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3812137472 0 0) 0x7f9050003530 con 0x7f905c03ba60 2024-03-19T01:23:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f9068d75640 1 --2- 172.21.15.7:0/3967025396 >> v2:172.21.15.167:3300/0 conn(0x7f905c03c080 0x7f905c036a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7fb41f59e640 1 -- start start 2024-03-19T01:23:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff401712640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3fc0b21a0 0x7ff3fc0b2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff401712640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff3fc145400 0x7ff3fc1458b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff401712640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3fc0b2d50 con 0x7ff3fc13ef80 2024-03-19T01:23:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff401712640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7ff3fc145df0 con 0x7ff3fc0b21a0 2024-03-19T01:23:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff401712640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7ff3fc070570 con 0x7ff3fc145400 2024-03-19T01:23:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f42ffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f4406c360 0x7f0f440ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f42ffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f4406c360 0x7f0f440ab270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45362/0 (socket says 172.21.15.7:45362) 2024-03-19T01:23:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f42ffd640 1 -- 172.21.15.7:0/882451841 learned_addr learned my addr 172.21.15.7:0/882451841 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f437fe640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 0x7f0f441e7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff3fa7fc640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff3fc145400 0x7ff3fc1458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff3fa7fc640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff3fc145400 0x7ff3fc1458b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45370/0 (socket says 172.21.15.7:45370) 2024-03-19T01:23:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff3fa7fc640 1 -- 172.21.15.7:0/3711626745 learned_addr learned my addr 172.21.15.7:0/3711626745 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7fb41f59e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb4180c3580 0x7fb4180c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f437fe640 1 -- 172.21.15.7:0/882451841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f4406c360 msgr2=0x7f0f440ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f437fe640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f4406c360 0x7f0f440ab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f437fe640 1 -- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f440721a0 msgr2=0x7f0f441e48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f437fe640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f440721a0 0x7f0f441e48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f437fe640 1 -- 172.21.15.7:0/882451841 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f440aa320 con 0x7f0f44153f00 2024-03-19T01:23:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7fb41f59e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb418037440 con 0x7fb418037060 2024-03-19T01:23:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7fb41f59e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4180c9510 con 0x7fb4180394a0 2024-03-19T01:23:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7fb41f59e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fb4180c9880 con 0x7fb4180c3580 2024-03-19T01:23:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f42ffd640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f4406c360 0x7f0f440ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff3faffd640 1 --2- 172.21.15.7:0/3711626745 >> v2:172.21.15.7:3300/0 conn(0x7ff3fc0b21a0 0x7ff3fc0b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7fb41e59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb418037060 0x7fb418037ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43164/0 (socket says 172.21.15.7:43164) 2024-03-19T01:23:33.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7fb41e59c640 1 -- 172.21.15.7:0/2364861079 learned_addr learned my addr 172.21.15.7:0/2364861079 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:33.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f437fe640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 0x7f0f441e7720 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00eab0 tx=0x7f0f3c00ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29dc0aaee0 con 0x7f29dc06b520 2024-03-19T01:23:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29dc0a9d40 con 0x7f29dc06b520 2024-03-19T01:23:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29d97fa640 1 -- 172.21.15.7:0/3183462873 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c80b49a0 con 0x7f29dc06b520 2024-03-19T01:23:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29d97fa640 1 -- 172.21.15.7:0/3183462873 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c80b4d70 con 0x7f29dc06b520 2024-03-19T01:23:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f29dc075460 con 0x7f29dc06b520 2024-03-19T01:23:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7fb41ed9d640 1 --2- 172.21.15.7:0/2364861079 >> v2:172.21.15.7:3300/0 conn(0x7fb4180c3580 0x7fb4180c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f9069576640 1 --2- 172.21.15.7:0/3967025396 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c037c70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f90600097c0 tx=0x7f906001c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e822ca46e7a7c97 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f3b7fe640 1 -- 172.21.15.7:0/882451841 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c018020 con 0x7f0f44153f00 2024-03-19T01:23:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f44149040 con 0x7f0f44153f00 2024-03-19T01:23:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f440aaaa0 con 0x7f0f44153f00 2024-03-19T01:23:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f3b7fe640 1 -- 172.21.15.7:0/882451841 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f3c00bc00 con 0x7f0f44153f00 2024-03-19T01:23:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f3b7fe640 1 -- 172.21.15.7:0/882451841 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c014400 con 0x7f0f44153f00 2024-03-19T01:23:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f440aa4f0 con 0x7f0f44153f00 2024-03-19T01:23:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7fb41ed9d640 1 -- 172.21.15.7:0/2364861079 >> v1:172.21.15.167:6789/0 conn(0x7fb4180394a0 legacy=0x7fb4180c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7fb41ed9d640 1 -- 172.21.15.7:0/2364861079 >> v1:172.21.15.7:6789/0 conn(0x7fb418037060 legacy=0x7fb418037ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f29d97fa640 1 -- 172.21.15.7:0/3183462873 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f29c80b3380 con 0x7f29dc06b520 2024-03-19T01:23:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f29d97fa640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f29c0067660 0x7f29c0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f29d97fa640 1 -- 172.21.15.7:0/3183462873 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f29c80ee280 con 0x7f29dc06b520 2024-03-19T01:23:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f29dbfff640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f29c0067660 0x7f29c0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f29d97fa640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7f29c006f860 0x7f29c0071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f29d97fa640 1 -- 172.21.15.7:0/3183462873 --> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29c800af30 con 0x7f29c006f860 2024-03-19T01:23:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f29d97fa640 1 -- 172.21.15.7:0/3183462873 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f29c8125050 con 0x7f29dc06b520 2024-03-19T01:23:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f905bfff640 1 -- 172.21.15.7:0/3967025396 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90600b0070 con 0x7f905c03b680 2024-03-19T01:23:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f905bfff640 1 -- 172.21.15.7:0/3967025396 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90600ab400 con 0x7f905c03b680 2024-03-19T01:23:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f905bfff640 1 -- 172.21.15.7:0/3967025396 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906404fde0 con 0x7f905c03b680 2024-03-19T01:23:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 -- 172.21.15.7:0/3967025396 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 msgr2=0x7f905c037c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 --2- 172.21.15.7:0/3967025396 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c037c70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f90600097c0 tx=0x7f906001c330 comp rx=0 tx=0).stop 2024-03-19T01:23:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7fb41ed9d640 1 -- 172.21.15.7:0/2364861079 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb418038b80 con 0x7fb4180c3580 2024-03-19T01:23:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff3fa7fc640 1 -- 172.21.15.7:0/3711626745 >> v2:172.21.15.7:3300/0 conn(0x7ff3fc0b21a0 msgr2=0x7ff3fc0b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff3fa7fc640 1 --2- 172.21.15.7:0/3711626745 >> v2:172.21.15.7:3300/0 conn(0x7ff3fc0b21a0 0x7ff3fc0b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff3fa7fc640 1 -- 172.21.15.7:0/3711626745 >> v1:172.21.15.7:6789/0 conn(0x7ff3fc13ef80 legacy=0x7ff3fc13f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.456+0000 7ff3fa7fc640 1 -- 172.21.15.7:0/3711626745 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3fc0b2ac0 con 0x7ff3fc145400 2024-03-19T01:23:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 -- 172.21.15.7:0/3967025396 shutdown_connections 2024-03-19T01:23:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 --2- 172.21.15.7:0/3967025396 >> v2:172.21.15.167:3300/0 conn(0x7f905c03c080 0x7f905c036a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 --2- 172.21.15.7:0/3967025396 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c037c70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 -- 172.21.15.7:0/3967025396 >> 172.21.15.7:0/3967025396 conn(0x7f905c03eab0 msgr2=0x7f905c03eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7ff3fa7fc640 1 --2- 172.21.15.7:0/3711626745 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3fc145400 0x7ff3fc1458b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc073250 tx=0x7ff3ec01c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21619f8c9adda9a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 -- 172.21.15.7:0/3967025396 shutdown_connections 2024-03-19T01:23:33.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 -- 172.21.15.7:0/3967025396 wait complete. 2024-03-19T01:23:33.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f29dbfff640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f29c0067660 0x7f29c0069b20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f29dc148740 tx=0x7f29d0007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f29e15a8640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7f29c006f860 0x7f29c0071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f0f3b7fe640 1 -- 172.21.15.7:0/882451841 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0f3c01b020 con 0x7f0f44153f00 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f0f3b7fe640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0f24067660 0x7f0f24069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f0f3b7fe640 1 -- 172.21.15.7:0/882451841 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0f3c0863e0 con 0x7f0f44153f00 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f0f3b7fe640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f0f2406f800 0x7f0f24071c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f0f3b7fe640 1 -- 172.21.15.7:0/882451841 --> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f3c0020c0 con 0x7f0f2406f800 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f0f3b7fe640 1 -- 172.21.15.7:0/882451841 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0f3c086870 con 0x7f0f44153f00 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f0f42ffd640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0f24067660 0x7f0f24069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f0f3bfff640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f0f2406f800 0x7f0f24071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f0f3bfff640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f0f2406f800 0x7f0f24071c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 Processor -- start 2024-03-19T01:23:33.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7f906a578640 1 -- start start 2024-03-19T01:23:33.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.460+0000 7fb41ed9d640 1 --2- 172.21.15.7:0/2364861079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c3580 0x7fb4180c9d10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb4180c9a00 tx=0x7fb41001bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5fbd634d96b801f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f29e15a8640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7f29c006f860 0x7f29c0071c80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3f9ffb640 1 -- 172.21.15.7:0/3711626745 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0aa430 con 0x7ff3fc145400 2024-03-19T01:23:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3f9ffb640 1 -- 172.21.15.7:0/3711626745 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3ec0af070 con 0x7ff3fc145400 2024-03-19T01:23:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3f9ffb640 1 -- 172.21.15.7:0/3711626745 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0aa7d0 con 0x7ff3fc145400 2024-03-19T01:23:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f0f42ffd640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0f24067660 0x7f0f24069b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0f34000f70 tx=0x7f0f34007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f906a578640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c02feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f906a578640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f905c03ba60 0x7f905c0303f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f906a578640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905c03c080 0x7f905c175260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f906a578640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f905c0dbe30 con 0x7f905c03c080 2024-03-19T01:23:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f906a578640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f905c0dbcb0 con 0x7f905c03ba60 2024-03-19T01:23:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f906a578640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f905c0dbfb0 con 0x7f905c03b680 2024-03-19T01:23:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9069576640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c02feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9069576640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c02feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:54972/0 (socket says 172.21.15.7:54972) 2024-03-19T01:23:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9069576640 1 -- 172.21.15.7:0/2504780793 learned_addr learned my addr 172.21.15.7:0/2504780793 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f29d97fa640 1 -- 172.21.15.7:0/3183462873 <== osd.0 v2:172.21.15.7:6800/2974347525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f29c800af30 con 0x7f29c006f860 2024-03-19T01:23:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 -- 172.21.15.7:0/3711626745 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3fc145400 msgr2=0x7ff3fc1458b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 --2- 172.21.15.7:0/3711626745 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3fc145400 0x7ff3fc1458b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc073250 tx=0x7ff3ec01c4d0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 -- 172.21.15.7:0/3711626745 shutdown_connections 2024-03-19T01:23:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 --2- 172.21.15.7:0/3711626745 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3fc145400 0x7ff3fc1458b0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc073250 tx=0x7ff3ec01c4d0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 --2- 172.21.15.7:0/3711626745 >> v2:172.21.15.7:3300/0 conn(0x7ff3fc0b21a0 0x7ff3fc0b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 -- 172.21.15.7:0/3711626745 >> 172.21.15.7:0/3711626745 conn(0x7ff3fc0b5330 msgr2=0x7ff3fc0b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9069576640 1 -- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f905c03ba60 msgr2=0x7f905c0303f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T01:23:33.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9069576640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f905c03ba60 0x7f905c0303f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9069576640 1 -- 172.21.15.7:0/2504780793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905c03c080 msgr2=0x7f905c175260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9069576640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f905c03c080 0x7f905c175260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9069576640 1 -- 172.21.15.7:0/2504780793 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c00d200 con 0x7f905c03b680 2024-03-19T01:23:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 -- 172.21.15.7:0/3711626745 shutdown_connections 2024-03-19T01:23:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f0f3b7fe640 1 -- 172.21.15.7:0/882451841 <== osd.1 v2:172.21.15.7:6808/1455113783 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f3c0020c0 con 0x7f0f2406f800 2024-03-19T01:23:33.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 -- 172.21.15.7:0/3711626745 wait complete. 2024-03-19T01:23:33.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9069576640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c02feb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f906001a9e0 tx=0x7f90600190d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9059ffb640 1 -- 172.21.15.7:0/2504780793 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90600b8020 con 0x7f905c03b680 2024-03-19T01:23:33.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9059ffb640 1 -- 172.21.15.7:0/2504780793 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90600b4630 con 0x7f905c03b680 2024-03-19T01:23:33.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 Processor -- start 2024-03-19T01:23:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 -- start start 2024-03-19T01:23:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7f9059ffb640 1 -- 172.21.15.7:0/2504780793 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90600b0040 con 0x7f905c03b680 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3fc0b21a0 0x7ff3fc1df4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3fc13ef80 0x7ff3fc134a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc12ead0 0x7ff3fc12ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc0afd30 con 0x7ff3fc12ead0 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc0b85d0 con 0x7ff3fc0b21a0 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff401712640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff3fc0afeb0 con 0x7ff3fc13ef80 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3fb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc12ead0 0x7ff3fc12ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3fb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc12ead0 0x7ff3fc12ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39892/0 (socket says 172.21.15.7:39892) 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3fb7fe640 1 -- 172.21.15.7:0/2698552575 learned_addr learned my addr 172.21.15.7:0/2698552575 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3fa7fc640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3fc13ef80 0x7ff3fc134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3faffd640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3fc0b21a0 0x7ff3fc1df4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3fb7fe640 1 -- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3fc13ef80 msgr2=0x7ff3fc134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3fb7fe640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3fc13ef80 0x7ff3fc134a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3fb7fe640 1 -- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3fc0b21a0 msgr2=0x7ff3fc1df4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3fb7fe640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3fc0b21a0 0x7ff3fc1df4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.464+0000 7ff3fb7fe640 1 -- 172.21.15.7:0/2698552575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3fc135380 con 0x7ff3fc12ead0 2024-03-19T01:23:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c00d4a0 con 0x7f905c03b680 2024-03-19T01:23:33.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c00c090 con 0x7f905c03b680 2024-03-19T01:23:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff3fb7fe640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc12ead0 0x7ff3fc12ef80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400b970 tx=0x7ff3f400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff3e37fe640 1 -- 172.21.15.7:0/2698552575 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4019070 con 0x7ff3fc12ead0 2024-03-19T01:23:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff3e37fe640 1 -- 172.21.15.7:0/2698552575 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3f400cb40 con 0x7ff3fc12ead0 2024-03-19T01:23:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3fc144680 con 0x7ff3fc12ead0 2024-03-19T01:23:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f905c0d1890 con 0x7f905c03b680 2024-03-19T01:23:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff3e37fe640 1 -- 172.21.15.7:0/2698552575 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4014440 con 0x7ff3fc12ead0 2024-03-19T01:23:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff3e37fe640 1 -- 172.21.15.7:0/2698552575 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff3f401c070 con 0x7ff3fc12ead0 2024-03-19T01:23:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3fc072be0 con 0x7ff3fc12ead0 2024-03-19T01:23:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff3e37fe640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff3e80676b0 0x7ff3e8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff3e37fe640 1 -- 172.21.15.7:0/2698552575 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff3f408a060 con 0x7ff3fc12ead0 2024-03-19T01:23:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff401712640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7ff3fc0b4100 0x7ff3fc0b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 --> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3fc13ea10 con 0x7ff3fc0b4100 2024-03-19T01:23:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff3fa7fc640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7ff3fc0b4100 0x7ff3fc0b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f9059ffb640 1 -- 172.21.15.7:0/2504780793 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f90600bb020 con 0x7f905c03b680 2024-03-19T01:23:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f9059ffb640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9048067690 0x7f9048069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f9059ffb640 1 -- 172.21.15.7:0/2504780793 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f906011fb50 con 0x7f905c03b680 2024-03-19T01:23:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f9059ffb640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7f904806f8b0 0x7f9048071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7ff3faffd640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff3e80676b0 0x7ff3e8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f9059ffb640 1 -- 172.21.15.7:0/2504780793 --> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90600a6ea0 con 0x7f904806f8b0 2024-03-19T01:23:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f9059ffb640 1 -- 172.21.15.7:0/2504780793 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f906011ff40 con 0x7f905c03b680 2024-03-19T01:23:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.468+0000 7f9068d75640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9048067690 0x7f9048069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7f9069d77640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7f904806f8b0 0x7f9048071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41d59a640 1 -- 172.21.15.7:0/2364861079 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4100a7620 con 0x7fb4180c3580 2024-03-19T01:23:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7ff3fa7fc640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7ff3fc0b4100 0x7ff3fc0b44e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41d59a640 1 -- 172.21.15.7:0/2364861079 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4100a77c0 con 0x7fb4180c3580 2024-03-19T01:23:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7ff3faffd640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff3e80676b0 0x7ff3e8069b70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0b95e0 tx=0x7ff3f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7f9068d75640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9048067690 0x7f9048069b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f905c031580 tx=0x7f9054008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7f9069d77640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7f904806f8b0 0x7f9048071cd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41f59e640 1 -- 172.21.15.7:0/2364861079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c3580 msgr2=0x7fb4180c9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41f59e640 1 --2- 172.21.15.7:0/2364861079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c3580 0x7fb4180c9d10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb4180c9a00 tx=0x7fb41001bf40 comp rx=0 tx=0).stop 2024-03-19T01:23:33.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41f59e640 1 -- 172.21.15.7:0/2364861079 shutdown_connections 2024-03-19T01:23:33.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41f59e640 1 --2- 172.21.15.7:0/2364861079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c3580 0x7fb4180c9d10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41f59e640 1 -- 172.21.15.7:0/2364861079 >> 172.21.15.7:0/2364861079 conn(0x7fb41803b9a0 msgr2=0x7fb41803bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41f59e640 1 -- 172.21.15.7:0/2364861079 shutdown_connections 2024-03-19T01:23:33.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41f59e640 1 -- 172.21.15.7:0/2364861079 wait complete. 2024-03-19T01:23:33.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7ff3e37fe640 1 -- 172.21.15.7:0/2698552575 <== osd.7 v2:172.21.15.167:6802/2393181242 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3fc13ea10 con 0x7ff3fc0b4100 2024-03-19T01:23:33.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7f9059ffb640 1 -- 172.21.15.7:0/2504780793 <== osd.4 v2:172.21.15.167:6800/2892054155 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f90600a6ea0 con 0x7f904806f8b0 2024-03-19T01:23:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41f59e640 1 Processor -- start 2024-03-19T01:23:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.472+0000 7fb41f59e640 1 -- start start 2024-03-19T01:23:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41f59e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb418037060 0x7fb41803c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41f59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180394a0 0x7fb41803e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41f59e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb4180c3580 0x7fb4180b9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41f59e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb418004b60 con 0x7fb4180394a0 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41f59e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb4180049e0 con 0x7fb418037060 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41f59e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb418004ce0 con 0x7fb4180c3580 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180394a0 0x7fb41803e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180394a0 0x7fb41803e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39900/0 (socket says 172.21.15.7:39900) 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41dd9b640 1 -- 172.21.15.7:0/595868657 learned_addr learned my addr 172.21.15.7:0/595868657 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41e59c640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb418037060 0x7fb41803c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41ed9d640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb4180c3580 0x7fb4180b9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41e59c640 1 -- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb4180c3580 msgr2=0x7fb4180b9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41e59c640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb4180c3580 0x7fb4180b9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41e59c640 1 -- 172.21.15.7:0/595868657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180394a0 msgr2=0x7fb41803e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41e59c640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180394a0 0x7fb41803e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41e59c640 1 -- 172.21.15.7:0/595868657 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4180c5bc0 con 0x7fb418037060 2024-03-19T01:23:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41dd9b640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180394a0 0x7fb41803e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41e59c640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb418037060 0x7fb41803c900 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb42007a5f0 tx=0x7fb420069570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f4030a50640 1 Processor -- start 2024-03-19T01:23:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb40effd640 1 -- 172.21.15.7:0/595868657 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420071b90 con 0x7fb418037060 2024-03-19T01:23:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f4030a50640 1 -- start start 2024-03-19T01:23:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4180c5e30 con 0x7fb418037060 2024-03-19T01:23:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4180c3af0 con 0x7fb418037060 2024-03-19T01:23:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f4030a50640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f402403b2e0 0x7f40240c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f402affd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f402403ba70 0x7f40240c6180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43176/0 (socket says 172.21.15.7:43176) 2024-03-19T01:23:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f402bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f402403b2e0 0x7f40240c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f402bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f402403b2e0 0x7f40240c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39910/0 (socket says 172.21.15.7:39910) 2024-03-19T01:23:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f402b7fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f402403a3e0 0x7f40240cced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:40754/0 (socket says 172.21.15.7:40754) 2024-03-19T01:23:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f402affd640 1 -- 172.21.15.7:0/683745425 learned_addr learned my addr 172.21.15.7:0/683745425 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f4030a50640 1 -- 172.21.15.7:0/683745425 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402403a7c0 con 0x7f402403ba70 2024-03-19T01:23:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f4030a50640 1 -- 172.21.15.7:0/683745425 --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40240cc6d0 con 0x7f402403a3e0 2024-03-19T01:23:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7f4030a50640 1 -- 172.21.15.7:0/683745425 --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f4024039d40 con 0x7f402403b2e0 2024-03-19T01:23:33.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb40effd640 1 -- 172.21.15.7:0/595868657 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb420071d30 con 0x7fb418037060 2024-03-19T01:23:33.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb40effd640 1 -- 172.21.15.7:0/595868657 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4200732c0 con 0x7fb418037060 2024-03-19T01:23:33.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.476+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb4180bb100 con 0x7fb418037060 2024-03-19T01:23:33.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 --> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f29dc149240 con 0x7f29c006f860 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb40effd640 1 -- 172.21.15.7:0/595868657 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb42006c050 con 0x7fb418037060 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb40effd640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb400067660 0x7fb400069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb40effd640 1 -- 172.21.15.7:0/595868657 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb420101470 con 0x7fb418037060 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb40effd640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7fb40006f8e0 0x7fb400071d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb41dd9b640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb400067660 0x7fb400069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb40effd640 1 -- 172.21.15.7:0/595868657 --> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4000723b0 con 0x7fb40006f8e0 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb40effd640 1 -- 172.21.15.7:0/595868657 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb420101860 con 0x7fb418037060 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb41ed9d640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7fb40006f8e0 0x7fb400071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7f402bfff640 1 -- 172.21.15.7:0/683745425 >> v1:172.21.15.167:6789/0 conn(0x7f402403a3e0 legacy=0x7f40240cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7f402bfff640 1 -- 172.21.15.7:0/683745425 >> v1:172.21.15.7:6789/0 conn(0x7f402403ba70 legacy=0x7f40240c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7f402bfff640 1 -- 172.21.15.7:0/683745425 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40240cc9b0 con 0x7f402403b2e0 2024-03-19T01:23:33.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb41dd9b640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb400067660 0x7fb400069b20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb418037b80 tx=0x7fb414009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7f29d97fa640 1 -- 172.21.15.7:0/3183462873 <== osd.0 v2:172.21.15.7:6800/2974347525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f29dc149240 con 0x7f29c006f860 2024-03-19T01:23:33.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.480+0000 7fb41ed9d640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7fb40006f8e0 0x7fb400071d00 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7f29c006f860 msgr2=0x7f29c0071c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7f29c006f860 0x7f29c0071c80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f29c0067660 msgr2=0x7f29c0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f29c0067660 0x7f29c0069b20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f29dc148740 tx=0x7f29d0007e40 comp rx=0 tx=0).stop 2024-03-19T01:23:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 msgr2=0x7f29dc0b2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 --2- 172.21.15.7:0/3183462873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29dc06b520 0x7f29dc0b2f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f29c801abd0 tx=0x7f29c801ad60 comp rx=0 tx=0).stop 2024-03-19T01:23:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e15a8640 1 -- 172.21.15.7:0/3183462873 reap_dead start 2024-03-19T01:23:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 shutdown_connections 2024-03-19T01:23:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 >> 172.21.15.7:0/3183462873 conn(0x7f29dc074130 msgr2=0x7f29dc14b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 shutdown_connections 2024-03-19T01:23:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f29e300b640 1 -- 172.21.15.7:0/3183462873 wait complete. 2024-03-19T01:23:33.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7fb40effd640 1 -- 172.21.15.7:0/595868657 <== osd.5 v2:172.21.15.167:6824/3760007162 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb4000723b0 con 0x7fb40006f8e0 2024-03-19T01:23:33.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f402bfff640 1 --2- 172.21.15.7:0/683745425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403b2e0 0x7f40240c6890 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4024039ec0 tx=0x7f402001c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d33c441ee781c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f402a7fc640 1 -- 172.21.15.7:0/683745425 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40200b0070 con 0x7f402403b2e0 2024-03-19T01:23:33.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f402a7fc640 1 -- 172.21.15.7:0/683745425 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40200ab600 con 0x7f402403b2e0 2024-03-19T01:23:33.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.484+0000 7f402a7fc640 1 -- 172.21.15.7:0/683745425 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40200ab9b0 con 0x7f402403b2e0 2024-03-19T01:23:33.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f4030a50640 1 -- 172.21.15.7:0/683745425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403b2e0 msgr2=0x7f40240c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f4030a50640 1 --2- 172.21.15.7:0/683745425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403b2e0 0x7f40240c6890 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4024039ec0 tx=0x7f402001c300 comp rx=0 tx=0).stop 2024-03-19T01:23:33.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f4030a50640 1 -- 172.21.15.7:0/683745425 shutdown_connections 2024-03-19T01:23:33.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f4030a50640 1 --2- 172.21.15.7:0/683745425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403b2e0 0x7f40240c6890 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f4030a50640 1 -- 172.21.15.7:0/683745425 >> 172.21.15.7:0/683745425 conn(0x7f402403eab0 msgr2=0x7f402403eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f4030a50640 1 -- 172.21.15.7:0/683745425 shutdown_connections 2024-03-19T01:23:33.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f2d595dc640 1 Processor -- start 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f2d595dc640 1 -- start start 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f2d595dc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2d540ab960 0x7f2d5414bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f2d595dc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b4a10 0x7f2d540b3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f2d595dc640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2d540b3dc0 0x7f2d54099660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f2d595dc640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f2d540b33f0 con 0x7f2d540ab960 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f2d595dc640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f2d5414c420 con 0x7f2d540b3dc0 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.488+0000 7f2d595dc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2d540abd40 con 0x7f2d540b4a10 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d53fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2d540ab960 0x7f2d5414bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d58ddb640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2d540b3dc0 0x7f2d54099660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d58ddb640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2d540b3dc0 0x7f2d54099660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45418/0 (socket says 172.21.15.7:45418) 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d58ddb640 1 -- 172.21.15.7:0/916358192 learned_addr learned my addr 172.21.15.7:0/916358192 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d58ddb640 1 -- 172.21.15.7:0/916358192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b4a10 msgr2=0x7f2d540b3880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:23:33.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d58ddb640 1 --2- 172.21.15.7:0/916358192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b4a10 0x7f2d540b3880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d58ddb640 1 -- 172.21.15.7:0/916358192 >> v2:172.21.15.7:3300/0 conn(0x7f2d540ab960 msgr2=0x7f2d5414bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d58ddb640 1 --2- 172.21.15.7:0/916358192 >> v2:172.21.15.7:3300/0 conn(0x7f2d540ab960 0x7f2d5414bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d58ddb640 1 -- 172.21.15.7:0/916358192 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5414c690 con 0x7f2d540b3dc0 2024-03-19T01:23:33.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f2d53fff640 1 --2- 172.21.15.7:0/916358192 >> v2:172.21.15.7:3300/0 conn(0x7f2d540ab960 0x7f2d5414bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:33.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f4030a50640 1 -- 172.21.15.7:0/683745425 wait complete. 2024-03-19T01:23:33.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f4030a50640 1 Processor -- start 2024-03-19T01:23:33.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.492+0000 7f4030a50640 1 -- start start 2024-03-19T01:23:33.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 --> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0f441e41b0 con 0x7f0f2406f800 2024-03-19T01:23:33.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f2d58ddb640 1 --2- 172.21.15.7:0/916358192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b3dc0 0x7f2d54099660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2d5409bbb0 tx=0x7f2d4c01ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bb3e52d9047ec50 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f4030a50640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f402403a3e0 0x7f40240c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f4030a50640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f402403b2e0 0x7f40240bb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f4030a50640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403ba70 0x7f40241699e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f4030a50640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f40240d8850 con 0x7f402403ba70 2024-03-19T01:23:33.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f4030a50640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f40240d86d0 con 0x7f402403a3e0 2024-03-19T01:23:33.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f4030a50640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f40240d89d0 con 0x7f402403b2e0 2024-03-19T01:23:33.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f2d52ffd640 1 -- 172.21.15.7:0/916358192 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0a57a0 con 0x7f2d540b3dc0 2024-03-19T01:23:33.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f0f3b7fe640 1 -- 172.21.15.7:0/882451841 <== osd.1 v2:172.21.15.7:6808/1455113783 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0f441e41b0 con 0x7f0f2406f800 2024-03-19T01:23:33.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f2d52ffd640 1 -- 172.21.15.7:0/916358192 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d4c0a5940 con 0x7f2d540b3dc0 2024-03-19T01:23:33.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f2d52ffd640 1 -- 172.21.15.7:0/916358192 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0a5d10 con 0x7f2d540b3dc0 2024-03-19T01:23:33.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f402bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403ba70 0x7f40241699e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f402bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403ba70 0x7f40241699e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39920/0 (socket says 172.21.15.7:39920) 2024-03-19T01:23:33.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f402bfff640 1 -- 172.21.15.7:0/1862443148 learned_addr learned my addr 172.21.15.7:0/1862443148 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f402bfff640 1 -- 172.21.15.7:0/1862443148 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f402403b2e0 msgr2=0x7f40240bb440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:23:33.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f402bfff640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f402403b2e0 0x7f40240bb440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f402bfff640 1 -- 172.21.15.7:0/1862443148 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f402403a3e0 msgr2=0x7f40240c9450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:23:33.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f402bfff640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f402403a3e0 0x7f40240c9450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f402bfff640 1 -- 172.21.15.7:0/1862443148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40240bd740 con 0x7f402403ba70 2024-03-19T01:23:33.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f2d595dc640 1 -- 172.21.15.7:0/916358192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b3dc0 msgr2=0x7f2d54099660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f2d595dc640 1 --2- 172.21.15.7:0/916358192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b3dc0 0x7f2d54099660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2d5409bbb0 tx=0x7f2d4c01ca10 comp rx=0 tx=0).stop 2024-03-19T01:23:33.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.496+0000 7f402bfff640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403ba70 0x7f40241699e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4020019f90 tx=0x7f4020019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f2d595dc640 1 -- 172.21.15.7:0/916358192 shutdown_connections 2024-03-19T01:23:33.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f2d595dc640 1 --2- 172.21.15.7:0/916358192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b3dc0 0x7f2d54099660 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f2d595dc640 1 --2- 172.21.15.7:0/916358192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b4a10 0x7f2d540b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f2d595dc640 1 --2- 172.21.15.7:0/916358192 >> v2:172.21.15.7:3300/0 conn(0x7f2d540ab960 0x7f2d5414bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f2d595dc640 1 -- 172.21.15.7:0/916358192 >> 172.21.15.7:0/916358192 conn(0x7f2d540b86b0 msgr2=0x7f2d540b8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4017fff640 1 -- 172.21.15.7:0/1862443148 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40200b9020 con 0x7f402403ba70 2024-03-19T01:23:33.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4017fff640 1 -- 172.21.15.7:0/1862443148 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40200aba70 con 0x7f402403ba70 2024-03-19T01:23:33.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40240bc810 con 0x7f402403ba70 2024-03-19T01:23:33.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4017fff640 1 -- 172.21.15.7:0/1862443148 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40200b0040 con 0x7f402403ba70 2024-03-19T01:23:33.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40240d07b0 con 0x7f402403ba70 2024-03-19T01:23:33.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4024005720 con 0x7f402403ba70 2024-03-19T01:23:33.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f2d595dc640 1 -- 172.21.15.7:0/916358192 shutdown_connections 2024-03-19T01:23:33.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f2d595dc640 1 -- 172.21.15.7:0/916358192 wait complete. 2024-03-19T01:23:33.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 --> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f905c0d1890 con 0x7f904806f8b0 2024-03-19T01:23:33.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f0f2406f800 msgr2=0x7f0f24071c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f0f43fff640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f0f2406f800 0x7f0f24071c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4017fff640 1 -- 172.21.15.7:0/1862443148 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f40200bc020 con 0x7f402403ba70 2024-03-19T01:23:33.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f9059ffb640 1 -- 172.21.15.7:0/2504780793 <== osd.4 v2:172.21.15.167:6800/2892054155 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f905c0d1890 con 0x7f904806f8b0 2024-03-19T01:23:33.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f2d595dc640 1 Processor -- start 2024-03-19T01:23:33.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4017fff640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f401c0677f0 0x7f401c069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7f904806f8b0 msgr2=0x7f9048071cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f906a578640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7f904806f8b0 0x7f9048071cd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9048067690 msgr2=0x7f9048069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f906a578640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9048067690 0x7f9048069b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f905c031580 tx=0x7f9054008040 comp rx=0 tx=0).stop 2024-03-19T01:23:33.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f402b7fe640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f401c0677f0 0x7f401c069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0f24067660 msgr2=0x7f0f24069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f0f43fff640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0f24067660 0x7f0f24069b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0f34000f70 tx=0x7f0f34007ed0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4017fff640 1 -- 172.21.15.7:0/1862443148 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f402011f7f0 con 0x7f402403ba70 2024-03-19T01:23:33.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f4017fff640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f401c06f9f0 0x7f401c071e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f4017fff640 1 -- 172.21.15.7:0/1862443148 --> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f401c0724c0 con 0x7f401c06f9f0 2024-03-19T01:23:33.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.500+0000 7f2d595dc640 1 -- start start 2024-03-19T01:23:33.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d595dc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540ab960 0x7f2d540ae320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d595dc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b3dc0 0x7f2d540afa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d595dc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d540b4a10 0x7f2d540affd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d595dc640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5414fac0 con 0x7f2d540b4a10 2024-03-19T01:23:33.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d595dc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5414f940 con 0x7f2d540ab960 2024-03-19T01:23:33.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d595dc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5414fc40 con 0x7f2d540b3dc0 2024-03-19T01:23:33.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 msgr2=0x7f0f441e7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f0f43fff640 1 --2- 172.21.15.7:0/882451841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f44153f00 0x7f0f441e7720 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00eab0 tx=0x7f0f3c00ef70 comp rx=0 tx=0).stop 2024-03-19T01:23:33.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f0f437fe640 1 -- 172.21.15.7:0/882451841 reap_dead start 2024-03-19T01:23:33.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f402affd640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f401c06f9f0 0x7f401c071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 shutdown_connections 2024-03-19T01:23:33.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 >> 172.21.15.7:0/882451841 conn(0x7f0f440b8c60 msgr2=0x7f0f44067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540ab960 0x7f2d540ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b3dc0 0x7f2d540afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b3dc0 0x7f2d540afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:55010/0 (socket says 172.21.15.7:55010) 2024-03-19T01:23:33.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540ab960 0x7f2d540ae320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45426/0 (socket says 172.21.15.7:45426) 2024-03-19T01:23:33.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d537fe640 1 -- 172.21.15.7:0/813007645 learned_addr learned my addr 172.21.15.7:0/813007645 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d58ddb640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d540b4a10 0x7f2d540affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 shutdown_connections 2024-03-19T01:23:33.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 msgr2=0x7f905c02feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f906a578640 1 --2- 172.21.15.7:0/2504780793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f905c03b680 0x7f905c02feb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f906001a9e0 tx=0x7f90600190d0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f9069d77640 1 -- 172.21.15.7:0/2504780793 reap_dead start 2024-03-19T01:23:33.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 shutdown_connections 2024-03-19T01:23:33.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 >> 172.21.15.7:0/2504780793 conn(0x7f905c03eab0 msgr2=0x7f905c03a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 shutdown_connections 2024-03-19T01:23:33.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d58ddb640 1 -- 172.21.15.7:0/813007645 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b3dc0 msgr2=0x7f2d540afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d58ddb640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b3dc0 0x7f2d540afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d58ddb640 1 -- 172.21.15.7:0/813007645 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540ab960 msgr2=0x7f2d540ae320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d58ddb640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540ab960 0x7f2d540ae320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d58ddb640 1 -- 172.21.15.7:0/813007645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d540706c0 con 0x7f2d540b4a10 2024-03-19T01:23:33.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f402affd640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f401c06f9f0 0x7f401c071e10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f906a578640 1 -- 172.21.15.7:0/2504780793 wait complete. 2024-03-19T01:23:33.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f402b7fe640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f401c0677f0 0x7f401c069cb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f402c064dc0 tx=0x7f402c06b010 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f0f43fff640 1 -- 172.21.15.7:0/882451841 wait complete. 2024-03-19T01:23:33.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d58ddb640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d540b4a10 0x7f2d540affd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c0ab000 tx=0x7f2d4c0197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d50ff9640 1 -- 172.21.15.7:0/813007645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0b8670 con 0x7f2d540b4a10 2024-03-19T01:23:33.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f4017fff640 1 -- 172.21.15.7:0/1862443148 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f40200ef840 con 0x7f402403ba70 2024-03-19T01:23:33.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d50ff9640 1 -- 172.21.15.7:0/813007645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d4c0be070 con 0x7f2d540b4a10 2024-03-19T01:23:33.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d50ff9640 1 -- 172.21.15.7:0/813007645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0b8a10 con 0x7f2d540b4a10 2024-03-19T01:23:33.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d54070930 con 0x7f2d540b4a10 2024-03-19T01:23:33.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d540b2120 con 0x7f2d540b4a10 2024-03-19T01:23:33.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d540752e0 con 0x7f2d540b4a10 2024-03-19T01:23:33.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.504+0000 7f4017fff640 1 -- 172.21.15.7:0/1862443148 <== osd.6 v2:172.21.15.167:6803/2582461346 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f401c0724c0 con 0x7f401c06f9f0 2024-03-19T01:23:33.529 INFO:teuthology.orchestra.run.smithi007.stdout:55834574853 2024-03-19T01:23:33.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-19T01:23:33.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.508+0000 7f2d50ff9640 1 -- 172.21.15.7:0/813007645 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2d4c0b8bb0 con 0x7f2d540b4a10 2024-03-19T01:23:33.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.508+0000 7f2d50ff9640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2d38067660 0x7f2d38069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.508+0000 7f2d50ff9640 1 -- 172.21.15.7:0/813007645 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2d4c121350 con 0x7f2d540b4a10 2024-03-19T01:23:33.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.508+0000 7f2d53fff640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2d38067660 0x7f2d38069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.508+0000 7f2d50ff9640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7f2d3806f8e0 0x7f2d38071d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.508+0000 7f2d50ff9640 1 -- 172.21.15.7:0/813007645 --> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d4c0a6e50 con 0x7f2d3806f8e0 2024-03-19T01:23:33.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.508+0000 7f2d50ff9640 1 -- 172.21.15.7:0/813007645 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2d4c121740 con 0x7f2d540b4a10 2024-03-19T01:23:33.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.508+0000 7f2d53fff640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2d38067660 0x7f2d38069b20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009950 tx=0x7f2d44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.512+0000 7f2d537fe640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7f2d3806f8e0 0x7f2d38071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.512+0000 7f2d537fe640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7f2d3806f8e0 0x7f2d38071d00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.516+0000 7f2d50ff9640 1 -- 172.21.15.7:0/813007645 <== osd.2 v2:172.21.15.7:6816/3939403805 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d4c0a6e50 con 0x7f2d3806f8e0 2024-03-19T01:23:33.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.516+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 --> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff3fc14f8c0 con 0x7ff3fc0b4100 2024-03-19T01:23:33.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.516+0000 7ff3e37fe640 1 -- 172.21.15.7:0/2698552575 <== osd.7 v2:172.21.15.167:6802/2393181242 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff3fc14f8c0 con 0x7ff3fc0b4100 2024-03-19T01:23:33.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7f37d9ab8640 1 Processor -- start 2024-03-19T01:23:33.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7f37d9ab8640 1 -- start start 2024-03-19T01:23:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7ff3fc0b4100 msgr2=0x7ff3fc0b44e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7ff401712640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7ff3fc0b4100 0x7ff3fc0b44e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff3e80676b0 msgr2=0x7ff3e8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7ff401712640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff3e80676b0 0x7ff3e8069b70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0b95e0 tx=0x7ff3f0008040 comp rx=0 tx=0).stop 2024-03-19T01:23:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc12ead0 msgr2=0x7ff3fc12ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7ff401712640 1 --2- 172.21.15.7:0/2698552575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc12ead0 0x7ff3fc12ef80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400b970 tx=0x7ff3f400be30 comp rx=0 tx=0).stop 2024-03-19T01:23:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7ff3fb7fe640 1 -- 172.21.15.7:0/2698552575 reap_dead start 2024-03-19T01:23:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 shutdown_connections 2024-03-19T01:23:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.520+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 >> 172.21.15.7:0/2698552575 conn(0x7ff3fc0b5330 msgr2=0x7ff3fc131c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 shutdown_connections 2024-03-19T01:23:33.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7ff401712640 1 -- 172.21.15.7:0/2698552575 wait complete. 2024-03-19T01:23:33.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d9ab8640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f37d406bed0 0x7f37d409d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d9ab8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d409c6d0 con 0x7f37d409b5e0 2024-03-19T01:23:33.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d9ab8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d406c2b0 con 0x7f37d406c6d0 2024-03-19T01:23:33.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d9ab8640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f37d406cb80 con 0x7f37d406bed0 2024-03-19T01:23:33.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 --> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb4180c65a0 con 0x7fb40006f8e0 2024-03-19T01:23:33.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d92b7640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f37d409b5e0 0x7f37d414de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43178/0 (socket says 172.21.15.7:43178) 2024-03-19T01:23:33.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d92b7640 1 -- 172.21.15.7:0/2426074395 learned_addr learned my addr 172.21.15.7:0/2426074395 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:33.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d8ab6640 1 --2- 172.21.15.7:0/2426074395 >> v2:172.21.15.167:3300/0 conn(0x7f37d406bed0 0x7f37d409d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7fb40effd640 1 -- 172.21.15.7:0/595868657 <== osd.5 v2:172.21.15.167:6824/3760007162 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb4180c65a0 con 0x7fb40006f8e0 2024-03-19T01:23:33.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d37fe640 1 -- 172.21.15.7:0/2426074395 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3096429481 0 0) 0x7f37d409c6d0 con 0x7f37d409b5e0 2024-03-19T01:23:33.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d8ab6640 1 -- 172.21.15.7:0/2426074395 >> v1:172.21.15.167:6789/0 conn(0x7f37d406c6d0 legacy=0x7f37d4099660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d8ab6640 1 -- 172.21.15.7:0/2426074395 >> v1:172.21.15.7:6789/0 conn(0x7f37d409b5e0 legacy=0x7f37d414de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7f37d8ab6640 1 -- 172.21.15.7:0/2426074395 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d406ce60 con 0x7f37d406bed0 2024-03-19T01:23:33.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7fb40006f8e0 msgr2=0x7fb400071d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.524+0000 7fb41f59e640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7fb40006f8e0 0x7fb400071d00 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb400067660 msgr2=0x7fb400069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7fb41f59e640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb400067660 0x7fb400069b20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb418037b80 tx=0x7fb414009040 comp rx=0 tx=0).stop 2024-03-19T01:23:33.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb418037060 msgr2=0x7fb41803c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7fb41f59e640 1 --2- 172.21.15.7:0/595868657 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb418037060 0x7fb41803c900 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb42007a5f0 tx=0x7fb420069570 comp rx=0 tx=0).stop 2024-03-19T01:23:33.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7fb41ed9d640 1 -- 172.21.15.7:0/595868657 reap_dead start 2024-03-19T01:23:33.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d8ab6640 1 --2- 172.21.15.7:0/2426074395 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37d406bed0 0x7f37d409d140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f37c80097c0 tx=0x7f37c801d610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad83c0b793b65ab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 shutdown_connections 2024-03-19T01:23:33.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 >> 172.21.15.7:0/595868657 conn(0x7fb41803b9a0 msgr2=0x7fb418005a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 shutdown_connections 2024-03-19T01:23:33.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7fb41f59e640 1 -- 172.21.15.7:0/595868657 wait complete. 2024-03-19T01:23:33.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d37fe640 1 -- 172.21.15.7:0/2426074395 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c80a76d0 con 0x7f37d406bed0 2024-03-19T01:23:33.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d37fe640 1 -- 172.21.15.7:0/2426074395 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37c80a7870 con 0x7f37d406bed0 2024-03-19T01:23:33.541 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2024-03-19T01:23:33.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-19T01:23:33.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d37fe640 1 -- 172.21.15.7:0/2426074395 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c80a7c40 con 0x7f37d406bed0 2024-03-19T01:23:33.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2426074395 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37d406bed0 msgr2=0x7f37d409d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d9ab8640 1 --2- 172.21.15.7:0/2426074395 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37d406bed0 0x7f37d409d140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f37c80097c0 tx=0x7f37c801d610 comp rx=0 tx=0).stop 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2426074395 shutdown_connections 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d9ab8640 1 --2- 172.21.15.7:0/2426074395 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37d406bed0 0x7f37d409d140 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2426074395 >> 172.21.15.7:0/2426074395 conn(0x7f37d40b86b0 msgr2=0x7f37d40b8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2426074395 shutdown_connections 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.528+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2426074395 wait complete. 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 Processor -- start 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 -- start start 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37d406bed0 0x7f37d40ab100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37d406c6d0 0x7f37d40a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37d409b5e0 0x7f37d41e34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f37d409da90 con 0x7f37d406c6d0 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f37d409d910 con 0x7f37d409b5e0 2024-03-19T01:23:33.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f37d4151d50 con 0x7f37d406bed0 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37d406c6d0 0x7f37d40a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37d406c6d0 0x7f37d40a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39938/0 (socket says 172.21.15.7:39938) 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d3fff640 1 -- 172.21.15.7:0/2892750859 learned_addr learned my addr 172.21.15.7:0/2892750859 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d8ab6640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37d406bed0 0x7f37d40ab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d92b7640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37d409b5e0 0x7f37d41e34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d3fff640 1 -- 172.21.15.7:0/2892750859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37d406bed0 msgr2=0x7f37d40ab100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d3fff640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37d406bed0 0x7f37d40ab100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d3fff640 1 -- 172.21.15.7:0/2892750859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37d409b5e0 msgr2=0x7f37d41e34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d3fff640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37d409b5e0 0x7f37d41e34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d3fff640 1 -- 172.21.15.7:0/2892750859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d40986b0 con 0x7f37d406c6d0 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d3fff640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37d406c6d0 0x7f37d40a2390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00c7c0 tx=0x7f37cc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d17fa640 1 -- 172.21.15.7:0/2892750859 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc003b80 con 0x7f37d406c6d0 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d17fa640 1 -- 172.21.15.7:0/2892750859 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37cc003d20 con 0x7f37d406c6d0 2024-03-19T01:23:33.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d17fa640 1 -- 172.21.15.7:0/2892750859 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc00aa30 con 0x7f37d406c6d0 2024-03-19T01:23:33.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d40aba80 con 0x7f37d406c6d0 2024-03-19T01:23:33.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d40a29a0 con 0x7f37d406c6d0 2024-03-19T01:23:33.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37d40752e0 con 0x7f37d406c6d0 2024-03-19T01:23:33.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.532+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 --> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f40240c9f50 con 0x7f401c06f9f0 2024-03-19T01:23:33.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4017fff640 1 -- 172.21.15.7:0/1862443148 <== osd.6 v2:172.21.15.167:6803/2582461346 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f40240c9f50 con 0x7f401c06f9f0 2024-03-19T01:23:33.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d17fa640 1 -- 172.21.15.7:0/2892750859 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f37cc00abd0 con 0x7f37d406c6d0 2024-03-19T01:23:33.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f401c06f9f0 msgr2=0x7f401c071e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f401c06f9f0 0x7f401c071e10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f401c0677f0 msgr2=0x7f401c069cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d17fa640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f37c0067660 0x7f37c0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f401c0677f0 0x7f401c069cb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f402c064dc0 tx=0x7f402c06b010 comp rx=0 tx=0).stop 2024-03-19T01:23:33.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403ba70 msgr2=0x7f40241699e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d17fa640 1 -- 172.21.15.7:0/2892750859 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f37cc084f00 con 0x7f37d406c6d0 2024-03-19T01:23:33.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 --2- 172.21.15.7:0/1862443148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f402403ba70 0x7f40241699e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4020019f90 tx=0x7f4020019100 comp rx=0 tx=0).stop 2024-03-19T01:23:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d17fa640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f37c006f860 0x7f37c0071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d17fa640 1 -- 172.21.15.7:0/2892750859 --> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37cc010e50 con 0x7f37c006f860 2024-03-19T01:23:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d17fa640 1 -- 172.21.15.7:0/2892750859 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f37cc08a060 con 0x7f37d406c6d0 2024-03-19T01:23:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d8ab6640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f37c0067660 0x7f37c0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d92b7640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f37c006f860 0x7f37c0071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f402bfff640 1 -- 172.21.15.7:0/1862443148 reap_dead start 2024-03-19T01:23:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 shutdown_connections 2024-03-19T01:23:33.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 >> 172.21.15.7:0/1862443148 conn(0x7f402403eab0 msgr2=0x7f4024005930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d8ab6640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f37c0067660 0x7f37c0069b20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f37c80b0000 tx=0x7f37c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d92b7640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f37c006f860 0x7f37c0071c80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 shutdown_connections 2024-03-19T01:23:33.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f4030a50640 1 -- 172.21.15.7:0/1862443148 wait complete. 2024-03-19T01:23:33.552 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2024-03-19T01:23:33.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-19T01:23:33.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.536+0000 7f37d17fa640 1 -- 172.21.15.7:0/2892750859 <== osd.3 v2:172.21.15.7:6824/1705011416 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37cc010e50 con 0x7f37c006f860 2024-03-19T01:23:33.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.540+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 --> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2d5414e310 con 0x7f2d3806f8e0 2024-03-19T01:23:33.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.540+0000 7f2d50ff9640 1 -- 172.21.15.7:0/813007645 <== osd.2 v2:172.21.15.7:6816/3939403805 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2d5414e310 con 0x7f2d3806f8e0 2024-03-19T01:23:33.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7f2d3806f8e0 msgr2=0x7f2d38071d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7f2d3806f8e0 0x7f2d38071d00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2d38067660 msgr2=0x7f2d38069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2d38067660 0x7f2d38069b20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009950 tx=0x7f2d44008040 comp rx=0 tx=0).stop 2024-03-19T01:23:33.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d540b4a10 msgr2=0x7f2d540affd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 --2- 172.21.15.7:0/813007645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d540b4a10 0x7f2d540affd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c0ab000 tx=0x7f2d4c0197b0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d58ddb640 1 -- 172.21.15.7:0/813007645 reap_dead start 2024-03-19T01:23:33.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 shutdown_connections 2024-03-19T01:23:33.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 >> 172.21.15.7:0/813007645 conn(0x7f2d540b86b0 msgr2=0x7f2d540b8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 shutdown_connections 2024-03-19T01:23:33.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.544+0000 7f2d595dc640 1 -- 172.21.15.7:0/813007645 wait complete. 2024-03-19T01:23:33.564 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2024-03-19T01:23:33.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-19T01:23:33.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 --> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37d406f150 con 0x7f37c006f860 2024-03-19T01:23:33.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d17fa640 1 -- 172.21.15.7:0/2892750859 <== osd.3 v2:172.21.15.7:6824/1705011416 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f37d406f150 con 0x7f37c006f860 2024-03-19T01:23:33.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f37c006f860 msgr2=0x7f37c0071c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d9ab8640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f37c006f860 0x7f37c0071c80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f37c0067660 msgr2=0x7f37c0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d9ab8640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f37c0067660 0x7f37c0069b20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f37c80b0000 tx=0x7f37c8008040 comp rx=0 tx=0).stop 2024-03-19T01:23:33.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37d406c6d0 msgr2=0x7f37d40a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d9ab8640 1 --2- 172.21.15.7:0/2892750859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37d406c6d0 0x7f37d40a2390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00c7c0 tx=0x7f37cc00cc80 comp rx=0 tx=0).stop 2024-03-19T01:23:33.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d92b7640 1 -- 172.21.15.7:0/2892750859 reap_dead start 2024-03-19T01:23:33.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 shutdown_connections 2024-03-19T01:23:33.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.564+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 >> 172.21.15.7:0/2892750859 conn(0x7f37d40b86b0 msgr2=0x7f37d40b3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.568+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 shutdown_connections 2024-03-19T01:23:33.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.568+0000 7f37d9ab8640 1 -- 172.21.15.7:0/2892750859 wait complete. 2024-03-19T01:23:33.572 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2024-03-19T01:23:33.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-19T01:23:33.574 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2024-03-19T01:23:33.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-19T01:23:33.579 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2024-03-19T01:23:33.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-19T01:23:33.599 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2024-03-19T01:23:33.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-19T01:23:33.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5ccc00b640 1 Processor -- start 2024-03-19T01:23:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5ccc00b640 1 -- start start 2024-03-19T01:23:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5ccc00b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f5cc40b31b0 0x7f5cc40b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5ccc00b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5cc40b3df0 0x7f5cc40b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5ccc00b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 0x7f5cc4099c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5ccc00b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc40b59e0 con 0x7f5cc40b3df0 2024-03-19T01:23:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5ccc00b640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc409c590 con 0x7f5cc40b31b0 2024-03-19T01:23:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5ccc00b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc40b3590 con 0x7f5cc40997d0 2024-03-19T01:23:33.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5cc95a6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5cc40b3df0 0x7f5cc40b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5cc95a6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5cc40b3df0 0x7f5cc40b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39944/0 (socket says 172.21.15.7:39944) 2024-03-19T01:23:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.712+0000 7f5cc95a6640 1 -- 172.21.15.7:0/199746773 learned_addr learned my addr 172.21.15.7:0/199746773 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.716+0000 7f5cc9da7640 1 --2- 172.21.15.7:0/199746773 >> v2:172.21.15.167:3300/0 conn(0x7f5cc40b31b0 0x7f5cc40b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.716+0000 7f5cca5a8640 1 --2- 172.21.15.7:0/199746773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 0x7f5cc4099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.716+0000 7f5cc95a6640 1 -- 172.21.15.7:0/199746773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 msgr2=0x7f5cc4099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.716+0000 7f5cc95a6640 1 --2- 172.21.15.7:0/199746773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 0x7f5cc4099c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.716+0000 7f5cc95a6640 1 -- 172.21.15.7:0/199746773 >> v2:172.21.15.167:3300/0 conn(0x7f5cc40b31b0 msgr2=0x7f5cc40b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.716+0000 7f5cc95a6640 1 --2- 172.21.15.7:0/199746773 >> v2:172.21.15.167:3300/0 conn(0x7f5cc40b31b0 0x7f5cc40b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.716+0000 7f5cc95a6640 1 -- 172.21.15.7:0/199746773 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc409c830 con 0x7f5cc40b3df0 2024-03-19T01:23:33.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.716+0000 7f5cca5a8640 1 --2- 172.21.15.7:0/199746773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 0x7f5cc4099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:33.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.716+0000 7f5cc95a6640 1 --2- 172.21.15.7:0/199746773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b3df0 0x7f5cc40b4260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5cc406f200 tx=0x7f5cbc01cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd9257c94f75092 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.720+0000 7f5cc8da5640 1 -- 172.21.15.7:0/199746773 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc0ab070 con 0x7f5cc40b3df0 2024-03-19T01:23:33.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.720+0000 7f5cc8da5640 1 -- 172.21.15.7:0/199746773 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cbc003ba0 con 0x7f5cc40b3df0 2024-03-19T01:23:33.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.720+0000 7f5cc8da5640 1 -- 172.21.15.7:0/199746773 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00be60 con 0x7f5cc40b3df0 2024-03-19T01:23:33.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.720+0000 7f5ccc00b640 1 -- 172.21.15.7:0/199746773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b3df0 msgr2=0x7f5cc40b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.720+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/199746773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b3df0 0x7f5cc40b4260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5cc406f200 tx=0x7f5cbc01cbb0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 -- 172.21.15.7:0/199746773 shutdown_connections 2024-03-19T01:23:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/199746773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 0x7f5cc4099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/199746773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b3df0 0x7f5cc40b4260 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/199746773 >> v2:172.21.15.167:3300/0 conn(0x7f5cc40b31b0 0x7f5cc40b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 -- 172.21.15.7:0/199746773 >> 172.21.15.7:0/199746773 conn(0x7f5cc409a2e0 msgr2=0x7f5cc409a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 -- 172.21.15.7:0/199746773 shutdown_connections 2024-03-19T01:23:33.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 -- 172.21.15.7:0/199746773 wait complete. 2024-03-19T01:23:33.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 Processor -- start 2024-03-19T01:23:33.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 -- start start 2024-03-19T01:23:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 0x7f5cc40ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b31b0 0x7f5cc41dcd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cc40b3df0 0x7f5cc41dd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc406c5c0 con 0x7f5cc40b31b0 2024-03-19T01:23:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc406c440 con 0x7f5cc40b3df0 2024-03-19T01:23:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5ccc00b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc406c740 con 0x7f5cc40997d0 2024-03-19T01:23:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5cc95a6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b31b0 0x7f5cc41dcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5cc95a6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b31b0 0x7f5cc41dcd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39946/0 (socket says 172.21.15.7:39946) 2024-03-19T01:23:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.724+0000 7f5cc95a6640 1 -- 172.21.15.7:0/459625802 learned_addr learned my addr 172.21.15.7:0/459625802 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.728+0000 7f5cc9da7640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 0x7f5cc40ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.728+0000 7f5cca5a8640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cc40b3df0 0x7f5cc41dd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.728+0000 7f5cc95a6640 1 -- 172.21.15.7:0/459625802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 msgr2=0x7f5cc40ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.728+0000 7f5cc95a6640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 0x7f5cc40ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.728+0000 7f5cc95a6640 1 -- 172.21.15.7:0/459625802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cc40b3df0 msgr2=0x7f5cc41dd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.728+0000 7f5cc95a6640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cc40b3df0 0x7f5cc41dd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.728+0000 7f5cc95a6640 1 -- 172.21.15.7:0/459625802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc40ae760 con 0x7f5cc40b31b0 2024-03-19T01:23:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.728+0000 7f5cc95a6640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b31b0 0x7f5cc41dcd40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc01d450 tx=0x7f5cbc019950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.732+0000 7f5cba7fc640 1 -- 172.21.15.7:0/459625802 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc0ab050 con 0x7f5cc40b31b0 2024-03-19T01:23:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.732+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc40aee20 con 0x7f5cc40b31b0 2024-03-19T01:23:33.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.732+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc40ae480 con 0x7f5cc40b31b0 2024-03-19T01:23:33.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.732+0000 7f5cba7fc640 1 -- 172.21.15.7:0/459625802 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cbc007aa0 con 0x7f5cc40b31b0 2024-03-19T01:23:33.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.732+0000 7f5cba7fc640 1 -- 172.21.15.7:0/459625802 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00be60 con 0x7f5cc40b31b0 2024-03-19T01:23:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.732+0000 7f5cba7fc640 1 -- 172.21.15.7:0/459625802 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5cbc004030 con 0x7f5cc40b31b0 2024-03-19T01:23:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.736+0000 7f5cba7fc640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5ca8067760 0x7f5ca8069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.736+0000 7f5cba7fc640 1 -- 172.21.15.7:0/459625802 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5cbc11f8f0 con 0x7f5cc40b31b0 2024-03-19T01:23:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.736+0000 7f5cc9da7640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5ca8067760 0x7f5ca8069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.736+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cac0052a0 con 0x7f5cc40b31b0 2024-03-19T01:23:33.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.740+0000 7f5cc9da7640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5ca8067760 0x7f5ca8069c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009ae0 tx=0x7f5cc0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7f5cba7fc640 1 -- 172.21.15.7:0/459625802 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5cbc0167a0 con 0x7f5cc40b31b0 2024-03-19T01:23:33.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9d2b9640 1 Processor -- start 2024-03-19T01:23:33.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9d2b9640 1 -- start start 2024-03-19T01:23:33.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9d2b9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fee9806c150 0x7fee980b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9d2b9640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fee98143ce0 0x7fee980b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9d2b9640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee981434e0 con 0x7fee9806c530 2024-03-19T01:23:33.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9d2b9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fee98144150 con 0x7fee9806c150 2024-03-19T01:23:33.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9d2b9640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fee981437c0 con 0x7fee98143ce0 2024-03-19T01:23:33.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9cab8640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fee98143ce0 0x7fee980b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9cab8640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fee98143ce0 0x7fee980b3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45476/0 (socket says 172.21.15.7:45476) 2024-03-19T01:23:33.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9cab8640 1 -- 172.21.15.7:0/517938635 learned_addr learned my addr 172.21.15.7:0/517938635 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee97fff640 1 --2- 172.21.15.7:0/517938635 >> v2:172.21.15.7:3300/0 conn(0x7fee9806c150 0x7fee980b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584bfff640 1 Processor -- start 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584bfff640 1 -- start start 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f584c0b1ed0 0x7f584c145ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584bfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c148590 0x7f584c0719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584bfff640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584c145130 con 0x7f584c06ef40 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584bfff640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f584c148a00 con 0x7f584c0b1ed0 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584bfff640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f584c1453a0 con 0x7f584c148590 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f584c0b1ed0 0x7f584c145ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584affd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f584c0b1ed0 0x7f584c145ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39972/0 (socket says 172.21.15.7:39972) 2024-03-19T01:23:33.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584affd640 1 -- 172.21.15.7:0/4074468048 learned_addr learned my addr 172.21.15.7:0/4074468048 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584affd640 1 -- 172.21.15.7:0/4074468048 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c148590 msgr2=0x7f584c0719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584affd640 1 --2- 172.21.15.7:0/4074468048 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c148590 0x7f584c0719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584affd640 1 -- 172.21.15.7:0/4074468048 >> v1:172.21.15.7:6789/0 conn(0x7f584c06ef40 legacy=0x7f584c072570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7f0656f99640 1 Processor -- start 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7f0656f99640 1 -- start start 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f0656f99640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0650145d10 0x7f065014ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f0656f99640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0650072bc0 0x7f065014bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f0656f99640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06500b1e00 con 0x7f06500b0460 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f0656f99640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f06500b0010 con 0x7f0650072bc0 2024-03-19T01:23:33.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f0656f99640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f06500b1160 con 0x7f0650145d10 2024-03-19T01:23:33.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f0654d35640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f06500b0460 0x7f0650145600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43186/0 (socket says 172.21.15.7:43186) 2024-03-19T01:23:33.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f0654d35640 1 -- 172.21.15.7:0/2756688316 learned_addr learned my addr 172.21.15.7:0/2756688316 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:33.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9cab8640 1 -- 172.21.15.7:0/517938635 >> v2:172.21.15.7:3300/0 conn(0x7fee9806c150 msgr2=0x7fee980b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9cab8640 1 --2- 172.21.15.7:0/517938635 >> v2:172.21.15.7:3300/0 conn(0x7fee9806c150 0x7fee980b2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9cab8640 1 -- 172.21.15.7:0/517938635 >> v1:172.21.15.167:6789/0 conn(0x7fee9806c530 legacy=0x7fee980b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.744+0000 7fee9cab8640 1 -- 172.21.15.7:0/517938635 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee98143190 con 0x7fee98143ce0 2024-03-19T01:23:33.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7fee9cab8640 1 --2- 172.21.15.7:0/517938635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee98143ce0 0x7fee980b3ab0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fee9806cb20 tx=0x7fee9001ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e75f0985835bd08e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7fee96ffd640 1 -- 172.21.15.7:0/517938635 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee900ad070 con 0x7fee98143ce0 2024-03-19T01:23:33.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7f584affd640 1 -- 172.21.15.7:0/4074468048 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c145680 con 0x7f584c0b1ed0 2024-03-19T01:23:33.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f0655536640 1 --2- 172.21.15.7:0/2756688316 >> v2:172.21.15.167:3300/0 conn(0x7f0650072bc0 0x7f065014bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7fee96ffd640 1 -- 172.21.15.7:0/517938635 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee900a7c10 con 0x7fee98143ce0 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.748+0000 7fee96ffd640 1 -- 172.21.15.7:0/517938635 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9000be60 con 0x7fee98143ce0 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 -- 172.21.15.7:0/517938635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee98143ce0 msgr2=0x7fee980b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 --2- 172.21.15.7:0/517938635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee98143ce0 0x7fee980b3ab0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fee9806cb20 tx=0x7fee9001ec30 comp rx=0 tx=0).stop 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 -- 172.21.15.7:0/517938635 shutdown_connections 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 --2- 172.21.15.7:0/517938635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee98143ce0 0x7fee980b3ab0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 --2- 172.21.15.7:0/517938635 >> v2:172.21.15.7:3300/0 conn(0x7fee9806c150 0x7fee980b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 -- 172.21.15.7:0/517938635 >> 172.21.15.7:0/517938635 conn(0x7fee980b8830 msgr2=0x7fee980b8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 -- 172.21.15.7:0/517938635 shutdown_connections 2024-03-19T01:23:33.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f0655536640 1 -- 172.21.15.7:0/2756688316 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0650145d10 msgr2=0x7f065014ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f0655536640 1 --2- 172.21.15.7:0/2756688316 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0650145d10 0x7f065014ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f0655536640 1 -- 172.21.15.7:0/2756688316 >> v1:172.21.15.7:6789/0 conn(0x7f06500b0460 legacy=0x7f0650145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f0655536640 1 -- 172.21.15.7:0/2756688316 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06500b0e30 con 0x7f0650072bc0 2024-03-19T01:23:33.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 -- 172.21.15.7:0/517938635 wait complete. 2024-03-19T01:23:33.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f584affd640 1 --2- 172.21.15.7:0/4074468048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c0b1ed0 0x7f584c145ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5844009e50 tx=0x7f584401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34df7db5a620f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f5849ffb640 1 -- 172.21.15.7:0/4074468048 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440a5770 con 0x7f584c0b1ed0 2024-03-19T01:23:33.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f5849ffb640 1 -- 172.21.15.7:0/4074468048 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58440a5910 con 0x7f584c0b1ed0 2024-03-19T01:23:33.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f5849ffb640 1 -- 172.21.15.7:0/4074468048 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440a5ce0 con 0x7f584c0b1ed0 2024-03-19T01:23:33.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 Processor -- start 2024-03-19T01:23:33.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f584bfff640 1 -- 172.21.15.7:0/4074468048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c0b1ed0 msgr2=0x7f584c145ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f584bfff640 1 --2- 172.21.15.7:0/4074468048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c0b1ed0 0x7f584c145ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5844009e50 tx=0x7f584401c8d0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f0655536640 1 --2- 172.21.15.7:0/2756688316 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0650072bc0 0x7f065014bf50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f06480097c0 tx=0x7f064801c2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e7d3c84e62dfb8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f584bfff640 1 -- 172.21.15.7:0/4074468048 shutdown_connections 2024-03-19T01:23:33.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f584bfff640 1 --2- 172.21.15.7:0/4074468048 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c148590 0x7f584c0719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f584bfff640 1 --2- 172.21.15.7:0/4074468048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c0b1ed0 0x7f584c145ba0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f584bfff640 1 -- 172.21.15.7:0/4074468048 >> 172.21.15.7:0/4074468048 conn(0x7f584c0b8830 msgr2=0x7f584c0b8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f584bfff640 1 -- 172.21.15.7:0/4074468048 shutdown_connections 2024-03-19T01:23:33.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 -- start start 2024-03-19T01:23:33.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee9806c150 0x7fee98145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee9806c530 0x7fee981e7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee98143ce0 0x7fee98139e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fee981506c0 con 0x7fee9806c150 2024-03-19T01:23:33.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fee98150540 con 0x7fee98143ce0 2024-03-19T01:23:33.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee9d2b9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fee98150840 con 0x7fee9806c530 2024-03-19T01:23:33.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fb8a5b9b640 1 Processor -- start 2024-03-19T01:23:33.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f064f7fe640 1 -- 172.21.15.7:0/2756688316 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06480b0070 con 0x7f0650072bc0 2024-03-19T01:23:33.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f064f7fe640 1 -- 172.21.15.7:0/2756688316 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06480ab600 con 0x7f0650072bc0 2024-03-19T01:23:33.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7f064f7fe640 1 -- 172.21.15.7:0/2756688316 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06480ab9d0 con 0x7f0650072bc0 2024-03-19T01:23:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee97fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee9806c150 0x7fee98145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee97fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee9806c150 0x7fee98145cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39984/0 (socket says 172.21.15.7:39984) 2024-03-19T01:23:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee97fff640 1 -- 172.21.15.7:0/3548599450 learned_addr learned my addr 172.21.15.7:0/3548599450 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fee977fe640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee9806c530 0x7fee981e7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee97fff640 1 -- 172.21.15.7:0/3548599450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee9806c530 msgr2=0x7fee981e7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee97fff640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee9806c530 0x7fee981e7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee97fff640 1 -- 172.21.15.7:0/3548599450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee98143ce0 msgr2=0x7fee98139e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:23:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee97fff640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee98143ce0 0x7fee98139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee97fff640 1 -- 172.21.15.7:0/3548599450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee98146f00 con 0x7fee9806c150 2024-03-19T01:23:33.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.752+0000 7fb8a5b9b640 1 -- start start 2024-03-19T01:23:33.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb8a5b9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb8a00b56f0 0x7fb8a01468b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb8a5b9b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fb8a0072680 0x7fb8a00b3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb8a5b9b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8a00b5ad0 con 0x7fb8a0072b10 2024-03-19T01:23:33.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb8a5b9b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fb8a00b09c0 con 0x7fb8a00b56f0 2024-03-19T01:23:33.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb8a5b9b640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fb8a00b1000 con 0x7fb8a0072680 2024-03-19T01:23:33.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f0656f99640 1 -- 172.21.15.7:0/2756688316 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0650072bc0 msgr2=0x7f065014bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f0656f99640 1 --2- 172.21.15.7:0/2756688316 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0650072bc0 0x7f065014bf50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f06480097c0 tx=0x7f064801c2c0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f584bfff640 1 -- 172.21.15.7:0/4074468048 wait complete. 2024-03-19T01:23:33.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f0656f99640 1 -- 172.21.15.7:0/2756688316 shutdown_connections 2024-03-19T01:23:33.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f0656f99640 1 --2- 172.21.15.7:0/2756688316 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0650072bc0 0x7f065014bf50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f0656f99640 1 --2- 172.21.15.7:0/2756688316 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0650145d10 0x7f065014ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f0656f99640 1 -- 172.21.15.7:0/2756688316 >> 172.21.15.7:0/2756688316 conn(0x7f06500b5330 msgr2=0x7f06500b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee97fff640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee9806c150 0x7fee98145cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00e930 tx=0x7fee8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee94ff9640 1 -- 172.21.15.7:0/3548599450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c017070 con 0x7fee9806c150 2024-03-19T01:23:33.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee94ff9640 1 -- 172.21.15.7:0/3548599450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee8c00ca00 con 0x7fee9806c150 2024-03-19T01:23:33.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee94ff9640 1 -- 172.21.15.7:0/3548599450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c00cdb0 con 0x7fee9806c150 2024-03-19T01:23:33.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb8a0072b10 0x7fb8a00b38a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43208/0 (socket says 172.21.15.7:43208) 2024-03-19T01:23:33.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89ffff640 1 -- 172.21.15.7:0/507934076 learned_addr learned my addr 172.21.15.7:0/507934076 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:33.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89f7fe640 1 --2- 172.21.15.7:0/507934076 >> v2:172.21.15.7:3300/0 conn(0x7fb8a00b56f0 0x7fb8a01468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1537005244 0 0) 0x7fb8a00b5ad0 con 0x7fb8a0072b10 2024-03-19T01:23:33.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb890003530 con 0x7fb8a0072b10 2024-03-19T01:23:33.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4144993080 0 0) 0x7fb890003530 con 0x7fb8a0072b10 2024-03-19T01:23:33.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 >> v2:172.21.15.167:3300/0 conn(0x7fb8a0072680 msgr2=0x7fb8a00b3360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:23:33.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 --2- 172.21.15.7:0/507934076 >> v2:172.21.15.167:3300/0 conn(0x7fb8a0072680 0x7fb8a00b3360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 >> v2:172.21.15.7:3300/0 conn(0x7fb8a00b56f0 msgr2=0x7fb8a01468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 --2- 172.21.15.7:0/507934076 >> v2:172.21.15.7:3300/0 conn(0x7fb8a00b56f0 0x7fb8a01468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f584bfff640 1 Processor -- start 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f584bfff640 1 -- start start 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f584bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c06ef40 0x7f584c0b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f584bfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c0b1ed0 0x7f584c146260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f584bfff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f584c148590 0x7f584c137510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f584bfff640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f584c0afdf0 con 0x7f584c06ef40 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f584bfff640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f584c0afc70 con 0x7f584c148590 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7f584bfff640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f584c0aff70 con 0x7f584c0b1ed0 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c06ef40 0x7f584c0b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c06ef40 0x7f584c0b67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40026/0 (socket says 172.21.15.7:40026) 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584affd640 1 -- 172.21.15.7:0/343569081 learned_addr learned my addr 172.21.15.7:0/343569081 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584a7fc640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c0b1ed0 0x7f584c146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584b7fe640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f584c148590 0x7f584c137510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584affd640 1 -- 172.21.15.7:0/343569081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c0b1ed0 msgr2=0x7f584c146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584affd640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c0b1ed0 0x7f584c146260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584affd640 1 -- 172.21.15.7:0/343569081 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f584c148590 msgr2=0x7f584c137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584affd640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f584c148590 0x7f584c137510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584affd640 1 -- 172.21.15.7:0/343569081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c1475f0 con 0x7f584c06ef40 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584a7fc640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c0b1ed0 0x7f584c146260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee981471e0 con 0x7fee9806c150 2024-03-19T01:23:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee981457b0 con 0x7fee9806c150 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a00b2450 con 0x7fb8a0072b10 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fb88c001660 con 0x7fb8a0072b10 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 >> v1:172.21.15.7:6789/0 conn(0x7fb8a0072b10 legacy=0x7fb8a00b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 --2- 172.21.15.7:0/507934076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb890003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 --2- 172.21.15.7:0/507934076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb890004810 0x7fb890004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89f7fe640 1 --2- 172.21.15.7:0/507934076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb890003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 --2- 172.21.15.7:0/507934076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890005320 0x7fb89000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a00b09c0 con 0x7fb890003a00 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a00b1000 con 0x7fb890005320 2024-03-19T01:23:33.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb89000c2e0 con 0x7fb890004810 2024-03-19T01:23:33.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89f7fe640 1 -- 172.21.15.7:0/507934076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb890004810 msgr2=0x7fb890004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:23:33.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89f7fe640 1 --2- 172.21.15.7:0/507934076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb890004810 0x7fb890004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89f7fe640 1 -- 172.21.15.7:0/507934076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890005320 msgr2=0x7fb89000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:23:33.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89f7fe640 1 --2- 172.21.15.7:0/507934076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890005320 0x7fb89000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89f7fe640 1 -- 172.21.15.7:0/507934076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a00b2450 con 0x7fb890003a00 2024-03-19T01:23:33.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.756+0000 7fb89ffff640 1 -- 172.21.15.7:0/507934076 reap_dead start 2024-03-19T01:23:33.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584affd640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c06ef40 0x7f584c0b67c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f584401af80 tx=0x7f5844019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7fee94ff9640 1 -- 172.21.15.7:0/3548599450 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fee8c01a070 con 0x7fee9806c150 2024-03-19T01:23:33.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f58337fe640 1 -- 172.21.15.7:0/343569081 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440bc020 con 0x7f584c06ef40 2024-03-19T01:23:33.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f58337fe640 1 -- 172.21.15.7:0/343569081 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58440b8880 con 0x7f584c06ef40 2024-03-19T01:23:33.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f58337fe640 1 -- 172.21.15.7:0/343569081 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440b8c70 con 0x7f584c06ef40 2024-03-19T01:23:33.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584bfff640 1 -- 172.21.15.7:0/343569081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584c0b71b0 con 0x7f584c06ef40 2024-03-19T01:23:33.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584bfff640 1 -- 172.21.15.7:0/343569081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f584c13f240 con 0x7f584c06ef40 2024-03-19T01:23:33.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7fee94ff9640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fee80067830 0x7fee80069cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7fee94ff9640 1 -- 172.21.15.7:0/3548599450 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fee8c085b90 con 0x7fee9806c150 2024-03-19T01:23:33.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f0656f99640 1 -- 172.21.15.7:0/2756688316 shutdown_connections 2024-03-19T01:23:33.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7fee977fe640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fee80067830 0x7fee80069cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f0656f99640 1 -- 172.21.15.7:0/2756688316 wait complete. 2024-03-19T01:23:33.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7fb89f7fe640 1 --2- 172.21.15.7:0/507934076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb890003e40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb89400b590 tx=0x7fb89401d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9cab4b1af23ae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7fee977fe640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fee80067830 0x7fee80069cf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fee98144ed0 tx=0x7fee88007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.760+0000 7f584bfff640 1 -- 172.21.15.7:0/343569081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58380052a0 con 0x7f584c06ef40 2024-03-19T01:23:33.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8940af020 con 0x7fb890003a00 2024-03-19T01:23:33.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8940a7630 con 0x7fb890003a00 2024-03-19T01:23:33.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb89e7fc640 1 -- 172.21.15.7:0/507934076 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89400b160 con 0x7fb890003a00 2024-03-19T01:23:33.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0656f99640 1 Processor -- start 2024-03-19T01:23:33.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0656f99640 1 -- start start 2024-03-19T01:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/507934076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 msgr2=0x7fb890003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb8a5b9b640 1 --2- 172.21.15.7:0/507934076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb890003e40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb89400b590 tx=0x7fb89401d8f0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f58337fe640 1 -- 172.21.15.7:0/343569081 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f58440bf020 con 0x7f584c06ef40 2024-03-19T01:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f58337fe640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58340677f0 0x7f5834069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f58337fe640 1 -- 172.21.15.7:0/343569081 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5844120af0 con 0x7f584c06ef40 2024-03-19T01:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0656f99640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0650072bc0 0x7f0650128450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0656f99640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b0460 0x7f06501336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0656f99640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0650145d10 0x7f06501269a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0656f99640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0650156b00 con 0x7f0650072bc0 2024-03-19T01:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0656f99640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0650156980 con 0x7f0650145d10 2024-03-19T01:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0656f99640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0650156c80 con 0x7f06500b0460 2024-03-19T01:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/507934076 shutdown_connections 2024-03-19T01:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb8a5b9b640 1 --2- 172.21.15.7:0/507934076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb890003e40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/507934076 >> 172.21.15.7:0/507934076 conn(0x7fb8a006caf0 msgr2=0x7fb8a0074130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f584a7fc640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58340677f0 0x7f5834069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f064ffff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b0460 0x7f06501336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f064ffff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b0460 0x7f06501336f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:55098/0 (socket says 172.21.15.7:55098) 2024-03-19T01:23:33.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f064ffff640 1 -- 172.21.15.7:0/4156771587 learned_addr learned my addr 172.21.15.7:0/4156771587 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f584a7fc640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58340677f0 0x7f5834069cb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f584c143ad0 tx=0x7f5840007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0655536640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0650145d10 0x7f06501269a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0654d35640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0650072bc0 0x7f0650128450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/507934076 shutdown_connections 2024-03-19T01:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0654d35640 1 -- 172.21.15.7:0/4156771587 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b0460 msgr2=0x7f06501336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0654d35640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b0460 0x7f06501336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0654d35640 1 -- 172.21.15.7:0/4156771587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0650145d10 msgr2=0x7f06501269a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0654d35640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0650145d10 0x7f06501269a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0654d35640 1 -- 172.21.15.7:0/4156771587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0650128990 con 0x7f0650072bc0 2024-03-19T01:23:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7f0654d35640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0650072bc0 0x7f0650128450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f064000cc50 tx=0x7f064000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.764+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/507934076 wait complete. 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb8a5b9b640 1 Processor -- start 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb8a5b9b640 1 -- start start 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb8a5b9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb8a01407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb8a5b9b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8a0070ef0 0x7fb8a013fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb8a5b9b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8a01ed0d0 0x7fb8a01ed580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb8a5b9b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a00b1490 con 0x7fb890003a00 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb8a5b9b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a00b2bc0 con 0x7fb8a0070ef0 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb8a5b9b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb8a01ecc40 con 0x7fb8a01ed0d0 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb8a01407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb8a01407a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40036/0 (socket says 172.21.15.7:40036) 2024-03-19T01:23:33.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89f7fe640 1 -- 172.21.15.7:0/3899264802 learned_addr learned my addr 172.21.15.7:0/3899264802 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89ffff640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8a01ed0d0 0x7fb8a01ed580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee981459b0 con 0x7fee9806c150 2024-03-19T01:23:33.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89f7fe640 1 -- 172.21.15.7:0/3899264802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8a01ed0d0 msgr2=0x7fb8a01ed580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89f7fe640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8a01ed0d0 0x7fb8a01ed580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89f7fe640 1 -- 172.21.15.7:0/3899264802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8a0070ef0 msgr2=0x7fb8a013fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89f7fe640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8a0070ef0 0x7fb8a013fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89f7fe640 1 -- 172.21.15.7:0/3899264802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a0071cc0 con 0x7fb890003a00 2024-03-19T01:23:33.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.768+0000 7fb89f7fe640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb8a01407a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb89401d720 tx=0x7fb890005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.772+0000 7f58337fe640 1 -- 172.21.15.7:0/343569081 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f58440c6030 con 0x7f584c06ef40 2024-03-19T01:23:33.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.772+0000 7f064d7fa640 1 -- 172.21.15.7:0/4156771587 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640017070 con 0x7f0650072bc0 2024-03-19T01:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.772+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06500b7a30 con 0x7f0650072bc0 2024-03-19T01:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.772+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06501260f0 con 0x7f0650072bc0 2024-03-19T01:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.772+0000 7fb88bfff640 1 -- 172.21.15.7:0/3899264802 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8940b8070 con 0x7fb890003a00 2024-03-19T01:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.772+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a0071390 con 0x7fb890003a00 2024-03-19T01:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.772+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a01f0010 con 0x7fb890003a00 2024-03-19T01:23:33.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0650075460 con 0x7f0650072bc0 2024-03-19T01:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea837c640 1 Processor -- start 2024-03-19T01:23:33.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea837c640 1 -- start start 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea837c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fce98037340 0x7fce98039770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea837c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 0x7fce980c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea837c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce980cb220 con 0x7fce98038e10 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea837c640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fce98037720 con 0x7fce98037340 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea837c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fce980c4a20 con 0x7fce980c40b0 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea6118640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fce98037340 0x7fce98039770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea6919640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 0x7fce980c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea5917640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fce98038e10 0x7fce980cab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:40780/0 (socket says 172.21.15.7:40780) 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea6118640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fce98037340 0x7fce98039770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45556/0 (socket says 172.21.15.7:45556) 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.776+0000 7fcea5917640 1 -- 172.21.15.7:0/4155217420 learned_addr learned my addr 172.21.15.7:0/4155217420 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:33.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8a00752e0 con 0x7fb890003a00 2024-03-19T01:23:33.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7fcea6118640 1 -- 172.21.15.7:0/4155217420 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 msgr2=0x7fce980c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7fcea6118640 1 --2- 172.21.15.7:0/4155217420 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 0x7fce980c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7fcea6118640 1 -- 172.21.15.7:0/4155217420 >> v1:172.21.15.167:6789/0 conn(0x7fce98038e10 legacy=0x7fce980cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7fcea6118640 1 -- 172.21.15.7:0/4155217420 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce98039dd0 con 0x7fce98037340 2024-03-19T01:23:33.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7fee94ff9640 1 -- 172.21.15.7:0/3548599450 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fee981459b0 con 0x7fee9806c150 2024-03-19T01:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7f064d7fa640 1 -- 172.21.15.7:0/4156771587 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f064000ad40 con 0x7f0650072bc0 2024-03-19T01:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7f064d7fa640 1 -- 172.21.15.7:0/4156771587 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064001c640 con 0x7f0650072bc0 2024-03-19T01:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7fb88bfff640 1 -- 172.21.15.7:0/3899264802 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8940a7630 con 0x7fb890003a00 2024-03-19T01:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.780+0000 7fcea6118640 1 --2- 172.21.15.7:0/4155217420 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce98037340 0x7fce98039770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0097c0 tx=0x7fce9c01c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4896741df925bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.784+0000 7fcea5116640 1 -- 172.21.15.7:0/4155217420 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0b0070 con 0x7fce98037340 2024-03-19T01:23:33.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.784+0000 7fcea5116640 1 -- 172.21.15.7:0/4155217420 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce9c0ab600 con 0x7fce98037340 2024-03-19T01:23:33.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.784+0000 7fcea5116640 1 -- 172.21.15.7:0/4155217420 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0ab9d0 con 0x7fce98037340 2024-03-19T01:23:33.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.784+0000 7fb88bfff640 1 -- 172.21.15.7:0/3899264802 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8940c2060 con 0x7fb890003a00 2024-03-19T01:23:33.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.784+0000 7fcea837c640 1 -- 172.21.15.7:0/4155217420 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce98037340 msgr2=0x7fce98039770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.784+0000 7fcea837c640 1 --2- 172.21.15.7:0/4155217420 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce98037340 0x7fce98039770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0097c0 tx=0x7fce9c01c380 comp rx=0 tx=0).stop 2024-03-19T01:23:33.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.784+0000 7f064d7fa640 1 -- 172.21.15.7:0/4156771587 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f064001a070 con 0x7f0650072bc0 2024-03-19T01:23:33.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.784+0000 7f064d7fa640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0630067690 0x7f0630069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.784+0000 7f064d7fa640 1 -- 172.21.15.7:0/4156771587 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0640085b50 con 0x7f0650072bc0 2024-03-19T01:23:33.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.788+0000 7f064ffff640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0630067690 0x7f0630069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.788+0000 7f064ffff640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0630067690 0x7f0630069b50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0650134930 tx=0x7f0638009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.788+0000 7fb88bfff640 1 -- 172.21.15.7:0/3899264802 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb8940bb070 con 0x7fb890003a00 2024-03-19T01:23:33.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.788+0000 7fb88bfff640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb884067600 0x7fb884069ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.788+0000 7fb88bfff640 1 -- 172.21.15.7:0/3899264802 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb89411fdb0 con 0x7fb890003a00 2024-03-19T01:23:33.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.788+0000 7fb89effd640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb884067600 0x7fb884069ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.788+0000 7fb89effd640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb884067600 0x7fb884069ac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb8a01411e0 tx=0x7fb898009250 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 -- 172.21.15.7:0/4155217420 shutdown_connections 2024-03-19T01:23:33.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 --2- 172.21.15.7:0/4155217420 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 0x7fce980c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 --2- 172.21.15.7:0/4155217420 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce98037340 0x7fce98039770 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 -- 172.21.15.7:0/4155217420 >> 172.21.15.7:0/4155217420 conn(0x7fce9803c6b0 msgr2=0x7fce9803cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 -- 172.21.15.7:0/4155217420 shutdown_connections 2024-03-19T01:23:33.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 -- 172.21.15.7:0/4155217420 wait complete. 2024-03-19T01:23:33.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 Processor -- start 2024-03-19T01:23:33.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 -- start start 2024-03-19T01:23:33.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7f064d7fa640 1 -- 172.21.15.7:0/4156771587 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0640021020 con 0x7f0650072bc0 2024-03-19T01:23:33.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce98037340 0x7fce980d41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce98038e10 0x7fce980bf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 0x7fce9816c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fce980d5470 con 0x7fce98037340 2024-03-19T01:23:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fce9803a5b0 con 0x7fce98038e10 2024-03-19T01:23:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea837c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fce980d55f0 con 0x7fce980c40b0 2024-03-19T01:23:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6118640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce98037340 0x7fce980d41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6118640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce98037340 0x7fce980d41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40050/0 (socket says 172.21.15.7:40050) 2024-03-19T01:23:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6118640 1 -- 172.21.15.7:0/227585957 learned_addr learned my addr 172.21.15.7:0/227585957 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6919640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 0x7fce9816c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea5917640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce98038e10 0x7fce980bf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6118640 1 -- 172.21.15.7:0/227585957 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 msgr2=0x7fce9816c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6118640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 0x7fce9816c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6118640 1 -- 172.21.15.7:0/227585957 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce98038e10 msgr2=0x7fce980bf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6118640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce98038e10 0x7fce980bf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6118640 1 -- 172.21.15.7:0/227585957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce980bfa70 con 0x7fce98037340 2024-03-19T01:23:33.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.792+0000 7fcea6118640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce98037340 0x7fce980d41d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0ae040 tx=0x7fce9c019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.800+0000 7fb88bfff640 1 -- 172.21.15.7:0/3899264802 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb894017600 con 0x7fb890003a00 2024-03-19T01:23:33.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.808+0000 7fce96ffd640 1 -- 172.21.15.7:0/227585957 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0b0040 con 0x7fce98037340 2024-03-19T01:23:33.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.808+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce980c4df0 con 0x7fce98037340 2024-03-19T01:23:33.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.808+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce980be410 con 0x7fce98037340 2024-03-19T01:23:33.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.812+0000 7fce96ffd640 1 -- 172.21.15.7:0/227585957 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce9c0b9070 con 0x7fce98037340 2024-03-19T01:23:33.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.812+0000 7fce96ffd640 1 -- 172.21.15.7:0/227585957 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0b4670 con 0x7fce98037340 2024-03-19T01:23:33.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.812+0000 7fce96ffd640 1 -- 172.21.15.7:0/227585957 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fce9c0b4810 con 0x7fce98037340 2024-03-19T01:23:33.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.812+0000 7fce96ffd640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fce84067760 0x7fce84069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.812+0000 7fce96ffd640 1 -- 172.21.15.7:0/227585957 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fce9c11f700 con 0x7fce98037340 2024-03-19T01:23:33.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.812+0000 7fcea5917640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fce84067760 0x7fce84069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.812+0000 7fcea5917640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fce84067760 0x7fce84069c20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fce98036c40 tx=0x7fce90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.820+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce880052a0 con 0x7fce98037340 2024-03-19T01:23:33.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.828+0000 7fce96ffd640 1 -- 172.21.15.7:0/227585957 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fce9c016090 con 0x7fce98037340 2024-03-19T01:23:33.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98c4c8640 1 Processor -- start 2024-03-19T01:23:33.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98c4c8640 1 -- start start 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98c4c8640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fb9840b21a0 0x7fb9840b2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98c4c8640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb984145400 0x7fb9841458b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98c4c8640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb98413ef80 0x7fb98413f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98c4c8640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fb984145df0 con 0x7fb984145400 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98c4c8640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fb984070570 con 0x7fb9840b21a0 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98c4c8640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb9840b2d50 con 0x7fb98413ef80 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb989a63640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb984145400 0x7fb9841458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb989a63640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb984145400 0x7fb9841458b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40056/0 (socket says 172.21.15.7:40056) 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98a264640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fb9840b21a0 0x7fb9840b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb989a63640 1 -- 172.21.15.7:0/2406677163 learned_addr learned my addr 172.21.15.7:0/2406677163 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb98aa65640 1 --2- 172.21.15.7:0/2406677163 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb98413ef80 0x7fb98413f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb989a63640 1 -- 172.21.15.7:0/2406677163 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb98413ef80 msgr2=0x7fb98413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb989a63640 1 --2- 172.21.15.7:0/2406677163 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb98413ef80 0x7fb98413f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb989a63640 1 -- 172.21.15.7:0/2406677163 >> v2:172.21.15.167:3300/0 conn(0x7fb9840b21a0 msgr2=0x7fb9840b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb989a63640 1 --2- 172.21.15.7:0/2406677163 >> v2:172.21.15.167:3300/0 conn(0x7fb9840b21a0 0x7fb9840b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.884+0000 7fb989a63640 1 -- 172.21.15.7:0/2406677163 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9840b2ac0 con 0x7fb984145400 2024-03-19T01:23:33.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.888+0000 7fb989a63640 1 --2- 172.21.15.7:0/2406677163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb984145400 0x7fb9841458b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb984067f80 tx=0x7fb97c01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb852e83db6bbf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb989262640 1 -- 172.21.15.7:0/2406677163 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c0a5790 con 0x7fb984145400 2024-03-19T01:23:33.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb989262640 1 -- 172.21.15.7:0/2406677163 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb97c0a5930 con 0x7fb984145400 2024-03-19T01:23:33.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb989262640 1 -- 172.21.15.7:0/2406677163 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c0a5d40 con 0x7fb984145400 2024-03-19T01:23:33.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb98c4c8640 1 -- 172.21.15.7:0/2406677163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb984145400 msgr2=0x7fb9841458b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/2406677163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb984145400 0x7fb9841458b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb984067f80 tx=0x7fb97c01ca00 comp rx=0 tx=0).stop 2024-03-19T01:23:33.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb98c4c8640 1 -- 172.21.15.7:0/2406677163 shutdown_connections 2024-03-19T01:23:33.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/2406677163 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb98413ef80 0x7fb98413f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/2406677163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb984145400 0x7fb9841458b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/2406677163 >> v2:172.21.15.167:3300/0 conn(0x7fb9840b21a0 0x7fb9840b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb98c4c8640 1 -- 172.21.15.7:0/2406677163 >> 172.21.15.7:0/2406677163 conn(0x7fb9840b5330 msgr2=0x7fb9840b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb98c4c8640 1 -- 172.21.15.7:0/2406677163 shutdown_connections 2024-03-19T01:23:33.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.896+0000 7fb98c4c8640 1 -- 172.21.15.7:0/2406677163 wait complete. 2024-03-19T01:23:33.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98c4c8640 1 Processor -- start 2024-03-19T01:23:33.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98c4c8640 1 -- start start 2024-03-19T01:23:33.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98c4c8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9840b21a0 0x7fb9841e74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98c4c8640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb98413ef80 0x7fb984140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98c4c8640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb984145400 0x7fb984140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98c4c8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb9840b85d0 con 0x7fb9840b21a0 2024-03-19T01:23:33.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98c4c8640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb98413f910 con 0x7fb98413ef80 2024-03-19T01:23:33.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98c4c8640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb984148f60 con 0x7fb984145400 2024-03-19T01:23:33.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98a264640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9840b21a0 0x7fb9841e74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98a264640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9840b21a0 0x7fb9841e74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40062/0 (socket says 172.21.15.7:40062) 2024-03-19T01:23:33.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98a264640 1 -- 172.21.15.7:0/4029034023 learned_addr learned my addr 172.21.15.7:0/4029034023 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98aa65640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb984145400 0x7fb984140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb989a63640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb98413ef80 0x7fb984140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98a264640 1 -- 172.21.15.7:0/4029034023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb984145400 msgr2=0x7fb984140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98a264640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb984145400 0x7fb984140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98a264640 1 -- 172.21.15.7:0/4029034023 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb98413ef80 msgr2=0x7fb984140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98a264640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb98413ef80 0x7fb984140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.900+0000 7fb98a264640 1 -- 172.21.15.7:0/4029034023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9841e8220 con 0x7fb9840b21a0 2024-03-19T01:23:33.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7fb98a264640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9840b21a0 0x7fb9841e74e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb98000e930 tx=0x7fb98000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f52057f4640 1 Processor -- start 2024-03-19T01:23:33.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f52057f4640 1 -- start start 2024-03-19T01:23:33.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f52057f4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5200072910 0x7f520014f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f52057f4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52001446f0 con 0x7f5200070880 2024-03-19T01:23:33.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f52057f4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52000b6880 con 0x7f52000b64a0 2024-03-19T01:23:33.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f52057f4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5200070d50 con 0x7f5200072910 2024-03-19T01:23:33.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f51feffd640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f52000b64a0 0x7f5200143fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:40794/0 (socket says 172.21.15.7:40794) 2024-03-19T01:23:33.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f51feffd640 1 -- 172.21.15.7:0/2237693154 learned_addr learned my addr 172.21.15.7:0/2237693154 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:33.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f51ff7fe640 1 --2- 172.21.15.7:0/2237693154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5200072910 0x7f520014f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f51ff7fe640 1 -- 172.21.15.7:0/2237693154 >> v1:172.21.15.167:6789/0 conn(0x7f52000b64a0 legacy=0x7f5200143fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f51ff7fe640 1 -- 172.21.15.7:0/2237693154 >> v1:172.21.15.7:6789/0 conn(0x7f5200070880 legacy=0x7f520013dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7f51ff7fe640 1 -- 172.21.15.7:0/2237693154 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52000b5770 con 0x7f5200072910 2024-03-19T01:23:33.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7fb97affd640 1 -- 172.21.15.7:0/4029034023 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980017070 con 0x7fb9840b21a0 2024-03-19T01:23:33.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb984142b40 con 0x7fb9840b21a0 2024-03-19T01:23:33.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.904+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb984144990 con 0x7fb9840b21a0 2024-03-19T01:23:33.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7fb97affd640 1 -- 172.21.15.7:0/4029034023 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb98000ca00 con 0x7fb9840b21a0 2024-03-19T01:23:33.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f51ff7fe640 1 --2- 172.21.15.7:0/2237693154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5200072910 0x7f520014f8b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5200070ed0 tx=0x7f51f801c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d730821206b471c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f51fdffb640 1 -- 172.21.15.7:0/2237693154 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f80b2070 con 0x7f5200072910 2024-03-19T01:23:33.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f51fdffb640 1 -- 172.21.15.7:0/2237693154 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51f80a5730 con 0x7f5200072910 2024-03-19T01:23:33.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f51fdffb640 1 -- 172.21.15.7:0/2237693154 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f80a5b00 con 0x7f5200072910 2024-03-19T01:23:33.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f52057f4640 1 -- 172.21.15.7:0/2237693154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5200072910 msgr2=0x7f520014f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f52057f4640 1 --2- 172.21.15.7:0/2237693154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5200072910 0x7f520014f8b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5200070ed0 tx=0x7f51f801c6d0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f52057f4640 1 -- 172.21.15.7:0/2237693154 shutdown_connections 2024-03-19T01:23:33.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f52057f4640 1 --2- 172.21.15.7:0/2237693154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5200072910 0x7f520014f8b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f52057f4640 1 -- 172.21.15.7:0/2237693154 >> 172.21.15.7:0/2237693154 conn(0x7f520006cd90 msgr2=0x7f520006d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f52057f4640 1 -- 172.21.15.7:0/2237693154 shutdown_connections 2024-03-19T01:23:33.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.908+0000 7f52057f4640 1 -- 172.21.15.7:0/2237693154 wait complete. 2024-03-19T01:23:33.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f52057f4640 1 Processor -- start 2024-03-19T01:23:33.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7fb97affd640 1 -- 172.21.15.7:0/4029034023 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98000cdb0 con 0x7fb9840b21a0 2024-03-19T01:23:33.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7fb97affd640 1 -- 172.21.15.7:0/4029034023 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb98001a070 con 0x7fb9840b21a0 2024-03-19T01:23:33.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7fb97affd640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb968067760 0x7fb968069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7fb989a63640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb968067760 0x7fb968069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f52057f4640 1 -- start start 2024-03-19T01:23:33.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7fb989a63640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb968067760 0x7fb968069c20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0ae000 tx=0x7fb97c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7fb97affd640 1 -- 172.21.15.7:0/4029034023 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb980085900 con 0x7fb9840b21a0 2024-03-19T01:23:33.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb96c0052a0 con 0x7fb9840b21a0 2024-03-19T01:23:33.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f52057f4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5200072910 0x7f52001d8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f52057f4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52000b64a0 0x7f52001cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f52057f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52001d9330 0x7f5200138400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f51feffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5200072910 0x7f52001d8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f51feffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5200072910 0x7f52001d8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45602/0 (socket says 172.21.15.7:45602) 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f51feffd640 1 -- 172.21.15.7:0/2245702870 learned_addr learned my addr 172.21.15.7:0/2245702870 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f51ff7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52001d9330 0x7f5200138400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f52000711e0 con 0x7f52001d9330 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.916+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f52000b5df0 con 0x7f5200072910 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.912+0000 7f51fe7fc640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52000b64a0 0x7f52001cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.916+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5200137460 con 0x7f52000b64a0 2024-03-19T01:23:33.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.916+0000 7f51ff7fe640 1 -- 172.21.15.7:0/2245702870 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52000b64a0 msgr2=0x7f52001cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.916+0000 7f51ff7fe640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52000b64a0 0x7f52001cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.916+0000 7f51ff7fe640 1 -- 172.21.15.7:0/2245702870 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5200072910 msgr2=0x7f52001d8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.916+0000 7f51ff7fe640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5200072910 0x7f52001d8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.916+0000 7f51ff7fe640 1 -- 172.21.15.7:0/2245702870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52001444a0 con 0x7f52001d9330 2024-03-19T01:23:33.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.916+0000 7f51feffd640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5200072910 0x7f52001d8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:33.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.916+0000 7f51ff7fe640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52001d9330 0x7f5200138400 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f51f80ac000 tx=0x7f51f8019470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.924+0000 7fb97affd640 1 -- 172.21.15.7:0/4029034023 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb9841e8220 con 0x7fb9840b21a0 2024-03-19T01:23:33.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.924+0000 7f51e77fe640 1 -- 172.21.15.7:0/2245702870 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f80bd030 con 0x7f52001d9330 2024-03-19T01:23:33.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.924+0000 7f51e77fe640 1 -- 172.21.15.7:0/2245702870 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51f80a5530 con 0x7f52001d9330 2024-03-19T01:23:33.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.924+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5200137740 con 0x7f52001d9330 2024-03-19T01:23:33.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.924+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5200137110 con 0x7f52001d9330 2024-03-19T01:23:33.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.928+0000 7f51e77fe640 1 -- 172.21.15.7:0/2245702870 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f80b2040 con 0x7f52001d9330 2024-03-19T01:23:33.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.928+0000 7f51e77fe640 1 -- 172.21.15.7:0/2245702870 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f51f80c0070 con 0x7f52001d9330 2024-03-19T01:23:33.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.928+0000 7f51e77fe640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f51ec067760 0x7f51ec069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.928+0000 7f51e77fe640 1 -- 172.21.15.7:0/2245702870 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f51f8120460 con 0x7f52001d9330 2024-03-19T01:23:33.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.928+0000 7f51feffd640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f51ec067760 0x7f51ec069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.928+0000 7f51feffd640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f51ec067760 0x7f51ec069c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f51f4000c10 tx=0x7f51f4007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.940+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e00052a0 con 0x7f52001d9330 2024-03-19T01:23:33.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:33.948+0000 7f51e77fe640 1 -- 172.21.15.7:0/2245702870 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f51f80c8020 con 0x7f52001d9330 2024-03-19T01:23:34.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.071+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5cac005590 con 0x7f5cc40b31b0 2024-03-19T01:23:34.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.071+0000 7f5cba7fc640 1 -- 172.21.15.7:0/459625802 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5cbc0eff40 con 0x7f5cc40b31b0 2024-03-19T01:23:34.076 INFO:teuthology.orchestra.run.smithi007.stdout:55834574852 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5ca8067760 msgr2=0x7f5ca8069c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5ca8067760 0x7f5ca8069c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009ae0 tx=0x7f5cc0009190 comp rx=0 tx=0).stop 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b31b0 msgr2=0x7f5cc41dcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b31b0 0x7f5cc41dcd40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc01d450 tx=0x7f5cbc019950 comp rx=0 tx=0).stop 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 shutdown_connections 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5ca8067760 0x7f5ca8069c20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cc40b3df0 0x7f5cc41dd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cc40b31b0 0x7f5cc41dcd40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 --2- 172.21.15.7:0/459625802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cc40997d0 0x7f5cc40ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 >> 172.21.15.7:0/459625802 conn(0x7f5cc409a2e0 msgr2=0x7f5cc40b8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:34.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 shutdown_connections 2024-03-19T01:23:34.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.087+0000 7f5ccc00b640 1 -- 172.21.15.7:0/459625802 wait complete. 2024-03-19T01:23:34.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2024-03-19T01:23:34.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.135+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb8a00b3e10 con 0x7fb890003a00 2024-03-19T01:23:34.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.135+0000 7fb88bfff640 1 -- 172.21.15.7:0/3899264802 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb8940f04f0 con 0x7fb890003a00 2024-03-19T01:23:34.142 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2024-03-19T01:23:34.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.135+0000 7f584bfff640 1 -- 172.21.15.7:0/343569081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5838005590 con 0x7f584c06ef40 2024-03-19T01:23:34.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.135+0000 7f58337fe640 1 -- 172.21.15.7:0/343569081 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f58440165e0 con 0x7f584c06ef40 2024-03-19T01:23:34.145 INFO:teuthology.orchestra.run.smithi007.stdout:55834574853 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 -- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58340677f0 msgr2=0x7f5834069cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58340677f0 0x7f5834069cb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f584c143ad0 tx=0x7f5840007e40 comp rx=0 tx=0).stop 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 -- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c06ef40 msgr2=0x7f584c0b67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c06ef40 0x7f584c0b67c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f584401af80 tx=0x7f5844019670 comp rx=0 tx=0).stop 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 -- 172.21.15.7:0/343569081 shutdown_connections 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58340677f0 0x7f5834069cb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f584c0b1ed0 0x7f584c146260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f584c148590 0x7f584c137510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 --2- 172.21.15.7:0/343569081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f584c06ef40 0x7f584c0b67c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 -- 172.21.15.7:0/343569081 >> 172.21.15.7:0/343569081 conn(0x7f584c0b8830 msgr2=0x7f584c0b8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:34.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 -- 172.21.15.7:0/343569081 shutdown_connections 2024-03-19T01:23:34.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.139+0000 7f58317fa640 1 -- 172.21.15.7:0/343569081 wait complete. 2024-03-19T01:23:34.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.143+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f065006b8b0 con 0x7f0650072bc0 2024-03-19T01:23:34.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.143+0000 7f064d7fa640 1 -- 172.21.15.7:0/4156771587 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f06400567c0 con 0x7f0650072bc0 2024-03-19T01:23:34.149 INFO:teuthology.orchestra.run.smithi007.stdout:55834574853 2024-03-19T01:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0630067690 msgr2=0x7f0630069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0630067690 0x7f0630069b50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0650134930 tx=0x7f0638009110 comp rx=0 tx=0).stop 2024-03-19T01:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0650072bc0 msgr2=0x7f0650128450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0650072bc0 0x7f0650128450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f064000cc50 tx=0x7f064000a430 comp rx=0 tx=0).stop 2024-03-19T01:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 shutdown_connections 2024-03-19T01:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0630067690 0x7f0630069b50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b0460 0x7f06501336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0650145d10 0x7f06501269a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 --2- 172.21.15.7:0/4156771587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0650072bc0 0x7f0650128450 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 >> 172.21.15.7:0/4156771587 conn(0x7f06500b5330 msgr2=0x7f0650144710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:34.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 shutdown_connections 2024-03-19T01:23:34.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.147+0000 7f0656f99640 1 -- 172.21.15.7:0/4156771587 wait complete. 2024-03-19T01:23:34.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb884067600 msgr2=0x7fb884069ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb884067600 0x7fb884069ac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb8a01411e0 tx=0x7fb898009250 comp rx=0 tx=0).stop 2024-03-19T01:23:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 msgr2=0x7fb8a01407a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb8a01407a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb89401d720 tx=0x7fb890005840 comp rx=0 tx=0).stop 2024-03-19T01:23:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 shutdown_connections 2024-03-19T01:23:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb884067600 0x7fb884069ac0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8a01ed0d0 0x7fb8a01ed580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8a0070ef0 0x7fb8a013fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 --2- 172.21.15.7:0/3899264802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb890003a00 0x7fb8a01407a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 >> 172.21.15.7:0/3899264802 conn(0x7fb8a006caf0 msgr2=0x7fb8a00b3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:34.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 shutdown_connections 2024-03-19T01:23:34.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.159+0000 7fb8a5b9b640 1 -- 172.21.15.7:0/3899264802 wait complete. 2024-03-19T01:23:34.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.163+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fce88005590 con 0x7fce98037340 2024-03-19T01:23:34.172 INFO:teuthology.orchestra.run.smithi007.stdout:55834574853 2024-03-19T01:23:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.163+0000 7fce96ffd640 1 -- 172.21.15.7:0/227585957 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fce9c0f03f0 con 0x7fce98037340 2024-03-19T01:23:34.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2024-03-19T01:23:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fce84067760 msgr2=0x7fce84069c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fce84067760 0x7fce84069c20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fce98036c40 tx=0x7fce90009040 comp rx=0 tx=0).stop 2024-03-19T01:23:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce98037340 msgr2=0x7fce980d41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce98037340 0x7fce980d41d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0ae040 tx=0x7fce9c019120 comp rx=0 tx=0).stop 2024-03-19T01:23:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 shutdown_connections 2024-03-19T01:23:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fce84067760 0x7fce84069c20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce980c40b0 0x7fce9816c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce98038e10 0x7fce980bf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 --2- 172.21.15.7:0/227585957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce98037340 0x7fce980d41d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 >> 172.21.15.7:0/227585957 conn(0x7fce9803c6b0 msgr2=0x7fce980c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 shutdown_connections 2024-03-19T01:23:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.167+0000 7fcea837c640 1 -- 172.21.15.7:0/227585957 wait complete. 2024-03-19T01:23:34.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2024-03-19T01:23:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.175+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fee980752e0 con 0x7fee9806c150 2024-03-19T01:23:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.175+0000 7fee94ff9640 1 -- 172.21.15.7:0/3548599450 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fee8c085020 con 0x7fee9806c150 2024-03-19T01:23:34.180 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2024-03-19T01:23:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fee80067830 msgr2=0x7fee80069cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fee80067830 0x7fee80069cf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fee98144ed0 tx=0x7fee88007e40 comp rx=0 tx=0).stop 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee9806c150 msgr2=0x7fee98145cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee9806c150 0x7fee98145cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00e930 tx=0x7fee8c00edf0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 shutdown_connections 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fee80067830 0x7fee80069cf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee9806c530 0x7fee981e7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee98143ce0 0x7fee98139e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 --2- 172.21.15.7:0/3548599450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee9806c150 0x7fee98145cd0 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00e930 tx=0x7fee8c00edf0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 >> 172.21.15.7:0/3548599450 conn(0x7fee980b8830 msgr2=0x7fee9814c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 shutdown_connections 2024-03-19T01:23:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.179+0000 7fee9d2b9640 1 -- 172.21.15.7:0/3548599450 wait complete. 2024-03-19T01:23:34.195 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-03-19T01:23:34.201 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-03-19T01:23:34.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2024-03-19T01:23:34.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb96c005590 con 0x7fb9840b21a0 2024-03-19T01:23:34.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb97affd640 1 -- 172.21.15.7:0/4029034023 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb980056060 con 0x7fb9840b21a0 2024-03-19T01:23:34.212 INFO:teuthology.orchestra.run.smithi007.stdout:55834574853 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb968067760 msgr2=0x7fb968069c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb968067760 0x7fb968069c20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0ae000 tx=0x7fb97c0a7070 comp rx=0 tx=0).stop 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9840b21a0 msgr2=0x7fb9841e74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9840b21a0 0x7fb9841e74e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb98000e930 tx=0x7fb98000edf0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 shutdown_connections 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb968067760 0x7fb968069c20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb984145400 0x7fb984140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb98413ef80 0x7fb984140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 --2- 172.21.15.7:0/4029034023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9840b21a0 0x7fb9841e74e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 >> 172.21.15.7:0/4029034023 conn(0x7fb9840b5330 msgr2=0x7fb9840b6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:34.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 shutdown_connections 2024-03-19T01:23:34.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.207+0000 7fb98c4c8640 1 -- 172.21.15.7:0/4029034023 wait complete. 2024-03-19T01:23:34.228 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-03-19T01:23:34.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f51e0005590 con 0x7f52001d9330 2024-03-19T01:23:34.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f51e77fe640 1 -- 172.21.15.7:0/2245702870 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f51f80164a0 con 0x7f52001d9330 2024-03-19T01:23:34.261 INFO:teuthology.orchestra.run.smithi007.stdout:55834574853 2024-03-19T01:23:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f51ec067760 msgr2=0x7f51ec069c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f51ec067760 0x7f51ec069c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f51f4000c10 tx=0x7f51f4007ef0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52001d9330 msgr2=0x7f5200138400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52001d9330 0x7f5200138400 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f51f80ac000 tx=0x7f51f8019470 comp rx=0 tx=0).stop 2024-03-19T01:23:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 shutdown_connections 2024-03-19T01:23:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f51ec067760 0x7f51ec069c20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52000b64a0 0x7f52001cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52001d9330 0x7f5200138400 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 --2- 172.21.15.7:0/2245702870 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5200072910 0x7f52001d8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:34.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.255+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 >> 172.21.15.7:0/2245702870 conn(0x7f520006cd90 msgr2=0x7f5200139e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:34.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.259+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 shutdown_connections 2024-03-19T01:23:34.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:34.259+0000 7f52057f4640 1 -- 172.21.15.7:0/2245702870 wait complete. 2024-03-19T01:23:34.283 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-03-19T01:23:35.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-19T01:23:35.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-19T01:23:35.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-19T01:23:35.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-19T01:23:35.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-19T01:23:35.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-19T01:23:35.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-19T01:23:35.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-19T01:23:35.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f745b6fe640 1 Processor -- start 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f745b6fe640 1 -- start start 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f745b6fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f74540aff90 0x7f7454146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f745b6fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f74540b0370 0x7f745413d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f745b6fe640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7454143590 con 0x7f7454143a90 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f745b6fe640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f74541475c0 con 0x7f74540b0370 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f745b6fe640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f74541478a0 con 0x7f74540aff90 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f7458c99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f74540b0370 0x7f745413d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f7458c99640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f74540b0370 0x7f745413d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40088/0 (socket says 172.21.15.7:40088) 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f7458c99640 1 -- 172.21.15.7:0/1515348028 learned_addr learned my addr 172.21.15.7:0/1515348028 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f745949a640 1 --2- 172.21.15.7:0/1515348028 >> v2:172.21.15.167:3300/0 conn(0x7f74540aff90 0x7f7454146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f7458c99640 1 -- 172.21.15.7:0/1515348028 >> v2:172.21.15.167:3300/0 conn(0x7f74540aff90 msgr2=0x7f7454146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f7458c99640 1 --2- 172.21.15.7:0/1515348028 >> v2:172.21.15.167:3300/0 conn(0x7f74540aff90 0x7f7454146fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f7458c99640 1 -- 172.21.15.7:0/1515348028 >> v1:172.21.15.167:6789/0 conn(0x7f7454143a90 legacy=0x7f7454143f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f7458c99640 1 -- 172.21.15.7:0/1515348028 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74540b2830 con 0x7f74540b0370 2024-03-19T01:23:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.307+0000 7f745949a640 1 --2- 172.21.15.7:0/1515348028 >> v2:172.21.15.167:3300/0 conn(0x7f74540aff90 0x7f7454146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7458c99640 1 --2- 172.21.15.7:0/1515348028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74540b0370 0x7f745413d660 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7454062f70 tx=0x7f744401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59364ae61206ba97 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fed702640 1 Processor -- start 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fed702640 1 -- start start 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fed702640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f7fe806d500 0x7f7fe80b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fed702640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 0x7f7fe814f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fed702640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fe806d8e0 con 0x7f7fe80ae340 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fed702640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f7fe814ec60 con 0x7f7fe806d500 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fed702640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe80afc60 con 0x7f7fe80b52a0 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fecf01640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 0x7f7fe814f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fe7fff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f7fe806d500 0x7f7fe80b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fe7fff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f7fe806d500 0x7f7fe80b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45614/0 (socket says 172.21.15.7:45614) 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fe7fff640 1 -- 172.21.15.7:0/3680090715 learned_addr learned my addr 172.21.15.7:0/3680090715 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fe7fff640 1 -- 172.21.15.7:0/3680090715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 msgr2=0x7f7fe814f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:35.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fe7fff640 1 --2- 172.21.15.7:0/3680090715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 0x7f7fe814f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fe7fff640 1 -- 172.21.15.7:0/3680090715 >> v1:172.21.15.167:6789/0 conn(0x7f7fe80ae340 legacy=0x7f7fe814f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:35.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.311+0000 7f7fe7fff640 1 -- 172.21.15.7:0/3680090715 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe80ae940 con 0x7f7fe806d500 2024-03-19T01:23:35.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f744bfff640 1 -- 172.21.15.7:0/1515348028 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74440b0070 con 0x7f74540b0370 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f744bfff640 1 -- 172.21.15.7:0/1515348028 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74440ab600 con 0x7f74540b0370 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f745b6fe640 1 -- 172.21.15.7:0/1515348028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74540b0370 msgr2=0x7f745413d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f745b6fe640 1 --2- 172.21.15.7:0/1515348028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74540b0370 0x7f745413d660 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7454062f70 tx=0x7f744401c320 comp rx=0 tx=0).stop 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f745b6fe640 1 -- 172.21.15.7:0/1515348028 shutdown_connections 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f745b6fe640 1 --2- 172.21.15.7:0/1515348028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74540b0370 0x7f745413d660 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f745b6fe640 1 --2- 172.21.15.7:0/1515348028 >> v2:172.21.15.167:3300/0 conn(0x7f74540aff90 0x7f7454146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f745b6fe640 1 -- 172.21.15.7:0/1515348028 >> 172.21.15.7:0/1515348028 conn(0x7f74540b5330 msgr2=0x7f74540b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f745b6fe640 1 -- 172.21.15.7:0/1515348028 shutdown_connections 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f745b6fe640 1 -- 172.21.15.7:0/1515348028 wait complete. 2024-03-19T01:23:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f7fe7fff640 1 --2- 172.21.15.7:0/3680090715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7fe806d500 0x7f7fe80b4d60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009ea0 tx=0x7f7fe001c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=204c5305648c32df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f7fe6ffd640 1 -- 172.21.15.7:0/3680090715 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe00a5590 con 0x7f7fe806d500 2024-03-19T01:23:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f745b6fe640 1 Processor -- start 2024-03-19T01:23:35.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f7fe6ffd640 1 -- 172.21.15.7:0/3680090715 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe00a5730 con 0x7f7fe806d500 2024-03-19T01:23:35.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.315+0000 7f7fe6ffd640 1 -- 172.21.15.7:0/3680090715 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe00a5b00 con 0x7f7fe806d500 2024-03-19T01:23:35.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 -- 172.21.15.7:0/3680090715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7fe806d500 msgr2=0x7f7fe80b4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 --2- 172.21.15.7:0/3680090715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7fe806d500 0x7f7fe80b4d60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009ea0 tx=0x7f7fe001c910 comp rx=0 tx=0).stop 2024-03-19T01:23:35.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 -- 172.21.15.7:0/3680090715 shutdown_connections 2024-03-19T01:23:35.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 --2- 172.21.15.7:0/3680090715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 0x7f7fe814f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 --2- 172.21.15.7:0/3680090715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7fe806d500 0x7f7fe80b4d60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 -- 172.21.15.7:0/3680090715 >> 172.21.15.7:0/3680090715 conn(0x7f7fe80b5780 msgr2=0x7f7fe80b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 -- 172.21.15.7:0/3680090715 shutdown_connections 2024-03-19T01:23:35.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745b6fe640 1 -- start start 2024-03-19T01:23:35.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745b6fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74540aff90 0x7f745413f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745b6fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7454143a90 0x7f74541403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745b6fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74541446f0 0x7f7454144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745b6fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f745413dd20 con 0x7f74541446f0 2024-03-19T01:23:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745b6fe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f745413dba0 con 0x7f74540aff90 2024-03-19T01:23:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745b6fe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f74540b5e20 con 0x7f7454143a90 2024-03-19T01:23:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745949a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74540aff90 0x7f745413f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7459c9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74541446f0 0x7f7454144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745949a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74540aff90 0x7f745413f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:45616/0 (socket says 172.21.15.7:45616) 2024-03-19T01:23:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745949a640 1 -- 172.21.15.7:0/909110550 learned_addr learned my addr 172.21.15.7:0/909110550 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7458c99640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7454143a90 0x7f74541403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745949a640 1 -- 172.21.15.7:0/909110550 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7454143a90 msgr2=0x7f74541403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745949a640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7454143a90 0x7f74541403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745949a640 1 -- 172.21.15.7:0/909110550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74541446f0 msgr2=0x7f7454144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745949a640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74541446f0 0x7f7454144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745949a640 1 -- 172.21.15.7:0/909110550 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7454140b60 con 0x7f74540aff90 2024-03-19T01:23:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 -- 172.21.15.7:0/3680090715 wait complete. 2024-03-19T01:23:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 Processor -- start 2024-03-19T01:23:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f745949a640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74540aff90 0x7f745413f1f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f744c00c2d0 tx=0x7f744c00c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 -- start start 2024-03-19T01:23:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7fe806d500 0x7f7fe8149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fe80ae340 0x7f7fe81323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 0x7f7fe81f0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe8156b70 con 0x7f7fe80ae340 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe81569f0 con 0x7f7fe806d500 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fed702640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe8156cf0 con 0x7f7fe80b52a0 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fe77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fe80ae340 0x7f7fe81323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fe77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fe80ae340 0x7f7fe81323c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40100/0 (socket says 172.21.15.7:40100) 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fe77fe640 1 -- 172.21.15.7:0/2941371034 learned_addr learned my addr 172.21.15.7:0/2941371034 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fe7fff640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7fe806d500 0x7f7fe8149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fecf01640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 0x7f7fe81f0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fe77fe640 1 -- 172.21.15.7:0/2941371034 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 msgr2=0x7f7fe81f0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.319+0000 7f7fe77fe640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 0x7f7fe81f0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fe77fe640 1 -- 172.21.15.7:0/2941371034 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7fe806d500 msgr2=0x7f7fe8149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fe77fe640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7fe806d500 0x7f7fe8149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fe77fe640 1 -- 172.21.15.7:0/2941371034 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe81ed1a0 con 0x7f7fe80ae340 2024-03-19T01:23:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fe77fe640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fe80ae340 0x7f7fe81323c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00c410 tx=0x7f7fdc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fe4ff9640 1 -- 172.21.15.7:0/2941371034 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc011790 con 0x7f7fe80ae340 2024-03-19T01:23:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fe4ff9640 1 -- 172.21.15.7:0/2941371034 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fdc017070 con 0x7f7fe80ae340 2024-03-19T01:23:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fe4ff9640 1 -- 172.21.15.7:0/2941371034 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc011bb0 con 0x7f7fe80ae340 2024-03-19T01:23:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fed702640 1 -- 172.21.15.7:0/2941371034 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe813b6a0 con 0x7f7fe80ae340 2024-03-19T01:23:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fed702640 1 -- 172.21.15.7:0/2941371034 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe8149f00 con 0x7f7fe80ae340 2024-03-19T01:23:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7449ffb640 1 -- 172.21.15.7:0/909110550 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c01a070 con 0x7f74540aff90 2024-03-19T01:23:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7449ffb640 1 -- 172.21.15.7:0/909110550 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f744c0039a0 con 0x7f74540aff90 2024-03-19T01:23:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7449ffb640 1 -- 172.21.15.7:0/909110550 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c003d70 con 0x7f74540aff90 2024-03-19T01:23:35.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101913640 1 Processor -- start 2024-03-19T01:23:35.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101913640 1 -- start start 2024-03-19T01:23:35.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7454140e40 con 0x7f74540aff90 2024-03-19T01:23:35.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f745413fcb0 con 0x7f74540aff90 2024-03-19T01:23:35.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101913640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4038850 0x7f80f40ca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101913640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80f4038cd0 0x7f80f40caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101913640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f4007150 con 0x7f80f4006d70 2024-03-19T01:23:35.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101913640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f80f403aa80 con 0x7f80f4038cd0 2024-03-19T01:23:35.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101913640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f80f40380f0 con 0x7f80f4038850 2024-03-19T01:23:35.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101112640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80f4038cd0 0x7f80f40caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101112640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80f4038cd0 0x7f80f40caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40110/0 (socket says 172.21.15.7:40110) 2024-03-19T01:23:35.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101112640 1 -- 172.21.15.7:0/4166327689 learned_addr learned my addr 172.21.15.7:0/4166327689 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f80fbfff640 1 --2- 172.21.15.7:0/4166327689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4038850 0x7f80f40ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f7fed702640 1 -- 172.21.15.7:0/2941371034 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe80683a0 con 0x7f7fe80ae340 2024-03-19T01:23:35.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7454075460 con 0x7f74540aff90 2024-03-19T01:23:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101112640 1 -- 172.21.15.7:0/4166327689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4038850 msgr2=0x7f80f40ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101112640 1 --2- 172.21.15.7:0/4166327689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4038850 0x7f80f40ca570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101112640 1 -- 172.21.15.7:0/4166327689 >> v1:172.21.15.167:6789/0 conn(0x7f80f4006d70 legacy=0x7f80f4036970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f8101112640 1 -- 172.21.15.7:0/4166327689 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f40caff0 con 0x7f80f4038cd0 2024-03-19T01:23:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.323+0000 7f80fbfff640 1 --2- 172.21.15.7:0/4166327689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4038850 0x7f80f40ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7fe4ff9640 1 -- 172.21.15.7:0/2941371034 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7fdc01c460 con 0x7f7fe80ae340 2024-03-19T01:23:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7fe4ff9640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7fd4067730 0x7f7fd4069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7fe4ff9640 1 -- 172.21.15.7:0/2941371034 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7fdc085540 con 0x7f7fe80ae340 2024-03-19T01:23:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7fe7fff640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7fd4067730 0x7f7fd4069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7449ffb640 1 -- 172.21.15.7:0/909110550 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f744c01d070 con 0x7f74540aff90 2024-03-19T01:23:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7449ffb640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7438067730 0x7f7438069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7449ffb640 1 -- 172.21.15.7:0/909110550 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f744c085920 con 0x7f74540aff90 2024-03-19T01:23:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7458c99640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7438067730 0x7f7438069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7fe7fff640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7fd4067730 0x7f7fd4069bf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7fe00ae000 tx=0x7f7fe00a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f7458c99640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7438067730 0x7f7438069bf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74440ad000 tx=0x7f74440a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.327+0000 7f8101112640 1 --2- 172.21.15.7:0/4166327689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f4038cd0 0x7f80f40caab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f80f403ac00 tx=0x7f80f001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effbf124cb3d6bff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.335+0000 7f7fe4ff9640 1 -- 172.21.15.7:0/2941371034 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7fe81ed1a0 con 0x7f7fe80ae340 2024-03-19T01:23:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.335+0000 7f80fb7fe640 1 -- 172.21.15.7:0/4166327689 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f00aa430 con 0x7f80f4038cd0 2024-03-19T01:23:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.335+0000 7f80fb7fe640 1 -- 172.21.15.7:0/4166327689 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f00af070 con 0x7f80f4038cd0 2024-03-19T01:23:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.335+0000 7f80fb7fe640 1 -- 172.21.15.7:0/4166327689 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f00aa830 con 0x7f80f4038cd0 2024-03-19T01:23:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 -- 172.21.15.7:0/4166327689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f4038cd0 msgr2=0x7f80f40caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 --2- 172.21.15.7:0/4166327689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f4038cd0 0x7f80f40caab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f80f403ac00 tx=0x7f80f001c4d0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 -- 172.21.15.7:0/4166327689 shutdown_connections 2024-03-19T01:23:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 --2- 172.21.15.7:0/4166327689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f4038cd0 0x7f80f40caab0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 --2- 172.21.15.7:0/4166327689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4038850 0x7f80f40ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 -- 172.21.15.7:0/4166327689 >> 172.21.15.7:0/4166327689 conn(0x7f80f4008710 msgr2=0x7f80f403eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 -- 172.21.15.7:0/4166327689 shutdown_connections 2024-03-19T01:23:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 -- 172.21.15.7:0/4166327689 wait complete. 2024-03-19T01:23:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f7449ffb640 1 -- 172.21.15.7:0/909110550 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f744c055fc0 con 0x7f74540aff90 2024-03-19T01:23:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 Processor -- start 2024-03-19T01:23:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 -- start start 2024-03-19T01:23:35.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4006d70 0x7f80f403df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f4038850 0x7f80f40c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80f40c9ca0 0x7f80f40cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f80f40ddac0 con 0x7f80f4038850 2024-03-19T01:23:35.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f80f40d3f10 con 0x7f80f40c9ca0 2024-03-19T01:23:35.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.339+0000 7f8101913640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f80f40ddc40 con 0x7f80f4006d70 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8100911640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4006d70 0x7f80f403df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8100911640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4006d70 0x7f80f403df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:55196/0 (socket says 172.21.15.7:55196) 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8100911640 1 -- 172.21.15.7:0/1706201328 learned_addr learned my addr 172.21.15.7:0/1706201328 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f80fbfff640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f4038850 0x7f80f40c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8101112640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80f40c9ca0 0x7f80f40cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8101112640 1 -- 172.21.15.7:0/1706201328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4006d70 msgr2=0x7f80f403df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8101112640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4006d70 0x7f80f403df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8101112640 1 -- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f4038850 msgr2=0x7f80f40c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8101112640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f4038850 0x7f80f40c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8101112640 1 -- 172.21.15.7:0/1706201328 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f40cd100 con 0x7f80f40c9ca0 2024-03-19T01:23:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8101112640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80f40c9ca0 0x7f80f40cbf20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f80f00ab000 tx=0x7f80f001ad10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f80f97fa640 1 -- 172.21.15.7:0/1706201328 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f00aa830 con 0x7f80f40c9ca0 2024-03-19T01:23:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f80f97fa640 1 -- 172.21.15.7:0/1706201328 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f0004310 con 0x7f80f40c9ca0 2024-03-19T01:23:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f80f97fa640 1 -- 172.21.15.7:0/1706201328 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f0003820 con 0x7f80f40c9ca0 2024-03-19T01:23:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f40cdf40 con 0x7f80f40c9ca0 2024-03-19T01:23:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.343+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f40cd3e0 con 0x7f80f40c9ca0 2024-03-19T01:23:35.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.347+0000 7f80f97fa640 1 -- 172.21.15.7:0/1706201328 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f80f00bb020 con 0x7f80f40c9ca0 2024-03-19T01:23:35.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.347+0000 7f80f97fa640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ec067850 0x7f80ec069d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.347+0000 7f80f97fa640 1 -- 172.21.15.7:0/1706201328 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f80f011fec0 con 0x7f80f40c9ca0 2024-03-19T01:23:35.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.347+0000 7f8100911640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ec067850 0x7f80ec069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.347+0000 7f8100911640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ec067850 0x7f80ec069d10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80fc000b70 tx=0x7f80fc08cf90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.355+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e00052a0 con 0x7f80f40c9ca0 2024-03-19T01:23:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.367+0000 7f80f97fa640 1 -- 172.21.15.7:0/1706201328 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f80f00c2060 con 0x7f80f40c9ca0 2024-03-19T01:23:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc770b1a640 1 Processor -- start 2024-03-19T01:23:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc770b1a640 1 -- start start 2024-03-19T01:23:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40be8af640 1 Processor -- start 2024-03-19T01:23:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc770b1a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7640d1130 0x7fc7640d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc770b1a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc764008340 con 0x7fc764007f60 2024-03-19T01:23:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc770b1a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7640072a0 con 0x7fc764039590 2024-03-19T01:23:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc770b1a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fc76403a110 con 0x7fc7640d1130 2024-03-19T01:23:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40be8af640 1 -- start start 2024-03-19T01:23:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40be8af640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40b0006d70 0x7f40b0036970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40be8af640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f40b0038850 0x7f40b00ca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40be8af640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b0007150 con 0x7f40b00caab0 2024-03-19T01:23:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40be8af640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f40b003aa80 con 0x7f40b0006d70 2024-03-19T01:23:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40be8af640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f40b00380f0 con 0x7f40b0038850 2024-03-19T01:23:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc769d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7640d1130 0x7fc7640d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc769d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7640d1130 0x7fc7640d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40132/0 (socket says 172.21.15.7:40132) 2024-03-19T01:23:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc769d9b640 1 -- 172.21.15.7:0/1094921588 learned_addr learned my addr 172.21.15.7:0/1094921588 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40bd8ad640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40b0006d70 0x7f40b0036970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40bd8ad640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40b0006d70 0x7f40b0036970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40140/0 (socket says 172.21.15.7:40140) 2024-03-19T01:23:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40bd8ad640 1 -- 172.21.15.7:0/1626279016 learned_addr learned my addr 172.21.15.7:0/1626279016 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40bd0ac640 1 --2- 172.21.15.7:0/1626279016 >> v2:172.21.15.167:3300/0 conn(0x7f40b0038850 0x7f40b00ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7f40bc8ab640 1 -- 172.21.15.7:0/1626279016 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3593409718 0 0) 0x7f40b0007150 con 0x7f40b00caab0 2024-03-19T01:23:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc769d9b640 1 -- 172.21.15.7:0/1094921588 >> v1:172.21.15.167:6789/0 conn(0x7fc764039590 legacy=0x7fc764039a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc769d9b640 1 -- 172.21.15.7:0/1094921588 >> v1:172.21.15.7:6789/0 conn(0x7fc764007f60 legacy=0x7fc7640092b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc769d9b640 1 -- 172.21.15.7:0/1094921588 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc764009a80 con 0x7fc7640d1130 2024-03-19T01:23:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.407+0000 7fc76959a640 1 -- 172.21.15.7:0/1094921588 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 69537905 0 0) 0x7fc764008340 con 0x7fc764007f60 2024-03-19T01:23:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.411+0000 7fc769d9b640 1 --2- 172.21.15.7:0/1094921588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 0x7fc7640d1570 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc760009870 tx=0x7fc76001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f53fa5b0787799 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.411+0000 7fc76959a640 1 -- 172.21.15.7:0/1094921588 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600b0070 con 0x7fc7640d1130 2024-03-19T01:23:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.411+0000 7fc76959a640 1 -- 172.21.15.7:0/1094921588 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7600ab600 con 0x7fc7640d1130 2024-03-19T01:23:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.411+0000 7fc76959a640 1 -- 172.21.15.7:0/1094921588 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600ab9d0 con 0x7fc7640d1130 2024-03-19T01:23:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.411+0000 7f40bc8ab640 1 -- 172.21.15.7:0/1626279016 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40a4003530 con 0x7f40b00caab0 2024-03-19T01:23:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.411+0000 7f40bd0ac640 1 -- 172.21.15.7:0/1626279016 >> v2:172.21.15.7:3300/0 conn(0x7f40b0006d70 msgr2=0x7f40b0036970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.411+0000 7f40bd0ac640 1 --2- 172.21.15.7:0/1626279016 >> v2:172.21.15.7:3300/0 conn(0x7f40b0006d70 0x7f40b0036970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.411+0000 7f40bd0ac640 1 -- 172.21.15.7:0/1626279016 >> v1:172.21.15.167:6789/0 conn(0x7f40b00caab0 legacy=0x7f40b00394a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.411+0000 7f40bd0ac640 1 -- 172.21.15.7:0/1626279016 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b003acd0 con 0x7f40b0038850 2024-03-19T01:23:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797b59e640 1 Processor -- start 2024-03-19T01:23:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797b59e640 1 -- start start 2024-03-19T01:23:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7fc770b1a640 1 -- 172.21.15.7:0/1094921588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 msgr2=0x7fc7640d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7fc770b1a640 1 --2- 172.21.15.7:0/1094921588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 0x7fc7640d1570 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc760009870 tx=0x7fc76001c460 comp rx=0 tx=0).stop 2024-03-19T01:23:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7fc770b1a640 1 -- 172.21.15.7:0/1094921588 shutdown_connections 2024-03-19T01:23:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7fc770b1a640 1 --2- 172.21.15.7:0/1094921588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 0x7fc7640d1570 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7fc770b1a640 1 -- 172.21.15.7:0/1094921588 >> 172.21.15.7:0/1094921588 conn(0x7fc76403c6e0 msgr2=0x7fc76403caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f40bd8ad640 1 --2- 172.21.15.7:0/1626279016 >> v2:172.21.15.7:3300/0 conn(0x7f40b0006d70 0x7f40b0036970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797b59e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f797402a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797b59e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7974036e20 con 0x7f79740cd910 2024-03-19T01:23:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797b59e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f797402aa10 con 0x7f79740c4850 2024-03-19T01:23:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797b59e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f797402ba80 con 0x7f79740371f0 2024-03-19T01:23:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797a59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f797402a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797a59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f797402a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:55204/0 (socket says 172.21.15.7:55204) 2024-03-19T01:23:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797a59c640 1 -- 172.21.15.7:0/2374865979 learned_addr learned my addr 172.21.15.7:0/2374865979 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797a59c640 1 -- 172.21.15.7:0/2374865979 >> v1:172.21.15.167:6789/0 conn(0x7f79740c4850 legacy=0x7f79740d7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797a59c640 1 -- 172.21.15.7:0/2374865979 >> v1:172.21.15.7:6789/0 conn(0x7f79740cd910 legacy=0x7f79740c4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f797a59c640 1 -- 172.21.15.7:0/2374865979 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7974039bf0 con 0x7f79740371f0 2024-03-19T01:23:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7fc770b1a640 1 -- 172.21.15.7:0/1094921588 shutdown_connections 2024-03-19T01:23:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f40bd0ac640 1 --2- 172.21.15.7:0/1626279016 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40b0038850 0x7f40b00ca570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f40b003f700 tx=0x7f40b401c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e24a7b7f86c03c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f40bc8ab640 1 -- 172.21.15.7:0/1626279016 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b40b2070 con 0x7f40b0038850 2024-03-19T01:23:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f40bc8ab640 1 -- 172.21.15.7:0/1626279016 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b40ad680 con 0x7f40b0038850 2024-03-19T01:23:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7f40bc8ab640 1 -- 172.21.15.7:0/1626279016 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b40ada50 con 0x7f40b0038850 2024-03-19T01:23:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 -- 172.21.15.7:0/1626279016 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40b0038850 msgr2=0x7f40b00ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 --2- 172.21.15.7:0/1626279016 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40b0038850 0x7f40b00ca570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f40b003f700 tx=0x7f40b401c3a0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.415+0000 7fc770b1a640 1 -- 172.21.15.7:0/1094921588 wait complete. 2024-03-19T01:23:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 -- 172.21.15.7:0/1626279016 shutdown_connections 2024-03-19T01:23:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 --2- 172.21.15.7:0/1626279016 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40b0038850 0x7f40b00ca570 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 --2- 172.21.15.7:0/1626279016 >> v2:172.21.15.7:3300/0 conn(0x7f40b0006d70 0x7f40b0036970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 -- 172.21.15.7:0/1626279016 >> 172.21.15.7:0/1626279016 conn(0x7f40b0008710 msgr2=0x7f40b003eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc770b1a640 1 Processor -- start 2024-03-19T01:23:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 -- 172.21.15.7:0/1626279016 shutdown_connections 2024-03-19T01:23:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 -- 172.21.15.7:0/1626279016 wait complete. 2024-03-19T01:23:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797a59c640 1 --2- 172.21.15.7:0/2374865979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f797402a4d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f797c0501c0 tx=0x7f797c098760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=494dd92506a0057c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797959a640 1 -- 172.21.15.7:0/2374865979 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c12d070 con 0x7f79740371f0 2024-03-19T01:23:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797959a640 1 -- 172.21.15.7:0/2374865979 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f797c120c00 con 0x7f79740371f0 2024-03-19T01:23:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797959a640 1 -- 172.21.15.7:0/2374865979 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c128660 con 0x7f79740371f0 2024-03-19T01:23:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc770b1a640 1 -- start start 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc770b1a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc764007f60 0x7fc7640ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc770b1a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc764039590 0x7fc7640c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc770b1a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 0x7fc7640cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc770b1a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc7640dee80 con 0x7fc7640d1130 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc770b1a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc7640ded00 con 0x7fc764039590 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc770b1a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc7640df000 con 0x7fc764007f60 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc76ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 0x7fc7640cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc76ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 0x7fc7640cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40144/0 (socket says 172.21.15.7:40144) 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc76ad9d640 1 -- 172.21.15.7:0/3651931065 learned_addr learned my addr 172.21.15.7:0/3651931065 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc769d9b640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc764039590 0x7fc7640c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc76a59c640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc764007f60 0x7fc7640ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 Processor -- start 2024-03-19T01:23:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797b59e640 1 -- 172.21.15.7:0/2374865979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 msgr2=0x7f797402a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797b59e640 1 --2- 172.21.15.7:0/2374865979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f797402a4d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f797c0501c0 tx=0x7f797c098760 comp rx=0 tx=0).stop 2024-03-19T01:23:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 -- start start 2024-03-19T01:23:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797b59e640 1 -- 172.21.15.7:0/2374865979 shutdown_connections 2024-03-19T01:23:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797b59e640 1 --2- 172.21.15.7:0/2374865979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f797402a4d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797b59e640 1 -- 172.21.15.7:0/2374865979 >> 172.21.15.7:0/2374865979 conn(0x7f797403a890 msgr2=0x7f797403aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797b59e640 1 -- 172.21.15.7:0/2374865979 shutdown_connections 2024-03-19T01:23:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc769d9b640 1 -- 172.21.15.7:0/3651931065 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc764007f60 msgr2=0x7fc7640ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc769d9b640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc764007f60 0x7fc7640ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc769d9b640 1 -- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 msgr2=0x7fc7640cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc769d9b640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 0x7fc7640cabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc769d9b640 1 -- 172.21.15.7:0/3651931065 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7640cc5d0 con 0x7fc764039590 2024-03-19T01:23:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc76ad9d640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 0x7fc7640cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7fc769d9b640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc764039590 0x7fc7640c9c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc7600ab000 tx=0x7fc76001aca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40b0006d70 0x7f40b00cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40b0038850 0x7f40b00afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40b00caab0 0x7f40b0175000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f40b00caf20 con 0x7f40b00caab0 2024-03-19T01:23:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f40b0036eb0 con 0x7f40b0038850 2024-03-19T01:23:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be8af640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f40b0038d00 con 0x7f40b0006d70 2024-03-19T01:23:35.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be0ae640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40b00caab0 0x7f40b0175000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be0ae640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40b00caab0 0x7f40b0175000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40146/0 (socket says 172.21.15.7:40146) 2024-03-19T01:23:35.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f40be0ae640 1 -- 172.21.15.7:0/2766020310 learned_addr learned my addr 172.21.15.7:0/2766020310 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40bd0ac640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40b0038850 0x7f40b00afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.419+0000 7f797b59e640 1 -- 172.21.15.7:0/2374865979 wait complete. 2024-03-19T01:23:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40be0ae640 1 -- 172.21.15.7:0/2766020310 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40b0006d70 msgr2=0x7f40b00cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40be0ae640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40b0006d70 0x7f40b00cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40be0ae640 1 -- 172.21.15.7:0/2766020310 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40b0038850 msgr2=0x7f40b00afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40be0ae640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40b0038850 0x7f40b00afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40be0ae640 1 -- 172.21.15.7:0/2766020310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b00bd3c0 con 0x7f40b00caab0 2024-03-19T01:23:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 Processor -- start 2024-03-19T01:23:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40be0ae640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40b00caab0 0x7f40b0175000 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f40b804f860 tx=0x7f40b8069570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40ae7fc640 1 -- 172.21.15.7:0/2766020310 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8071b30 con 0x7f40b00caab0 2024-03-19T01:23:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b00cd030 con 0x7f40b00caab0 2024-03-19T01:23:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b00bd0e0 con 0x7f40b00caab0 2024-03-19T01:23:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 -- start start 2024-03-19T01:23:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f79740cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79740c4850 0x7f79740b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79740cd910 0x7f79740cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f79740cf420 con 0x7f79740cd910 2024-03-19T01:23:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f79740375d0 con 0x7f79740c4850 2024-03-19T01:23:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f79740cf5a0 con 0x7f79740371f0 2024-03-19T01:23:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797a59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f79740cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797a59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f79740cd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:55242/0 (socket says 172.21.15.7:55242) 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797a59c640 1 -- 172.21.15.7:0/3551601885 learned_addr learned my addr 172.21.15.7:0/3551601885 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797ad9d640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79740cd910 0x7f79740cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f7979d9b640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79740c4850 0x7f79740b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797a59c640 1 -- 172.21.15.7:0/3551601885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79740c4850 msgr2=0x7f79740b6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797a59c640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79740c4850 0x7f79740b6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797a59c640 1 -- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79740cd910 msgr2=0x7f79740cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797a59c640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79740cd910 0x7f79740cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797a59c640 1 -- 172.21.15.7:0/3551601885 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797403c520 con 0x7f79740371f0 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797ad9d640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79740cd910 0x7f79740cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797a59c640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f79740cd2a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f797c000b70 tx=0x7f797c094b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f796affd640 1 -- 172.21.15.7:0/3551601885 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c12d050 con 0x7f79740371f0 2024-03-19T01:23:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f40ae7fc640 1 -- 172.21.15.7:0/2766020310 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b8071cd0 con 0x7f40b00caab0 2024-03-19T01:23:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f796affd640 1 -- 172.21.15.7:0/3551601885 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f797c12e040 con 0x7f79740371f0 2024-03-19T01:23:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f796affd640 1 -- 172.21.15.7:0/3551601885 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c134670 con 0x7f79740371f0 2024-03-19T01:23:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f797403c800 con 0x7f79740371f0 2024-03-19T01:23:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.423+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f797403b7f0 con 0x7f79740371f0 2024-03-19T01:23:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f40ae7fc640 1 -- 172.21.15.7:0/2766020310 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8073280 con 0x7f40b00caab0 2024-03-19T01:23:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f40ae7fc640 1 -- 172.21.15.7:0/2766020310 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f40b806c050 con 0x7f40b00caab0 2024-03-19T01:23:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79740052d0 con 0x7f79740371f0 2024-03-19T01:23:35.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f40ae7fc640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f409c067790 0x7f409c069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f40ae7fc640 1 -- 172.21.15.7:0/2766020310 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f40b8100fc0 con 0x7f40b00caab0 2024-03-19T01:23:35.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f796affd640 1 -- 172.21.15.7:0/3551601885 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f797c13b020 con 0x7f79740371f0 2024-03-19T01:23:35.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f40bd8ad640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f409c067790 0x7f409c069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f796affd640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f795c067660 0x7f795c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f796affd640 1 -- 172.21.15.7:0/3551601885 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f797c19c020 con 0x7f79740371f0 2024-03-19T01:23:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f7979d9b640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f795c067660 0x7f795c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f40bd8ad640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f409c067790 0x7f409c069c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009630 tx=0x7f40a8006ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.427+0000 7f7979d9b640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f795c067660 0x7f795c069b20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f797403bb60 tx=0x7f796c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.431+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a00052a0 con 0x7f40b00caab0 2024-03-19T01:23:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.435+0000 7f40ae7fc640 1 -- 172.21.15.7:0/2766020310 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f40b808f030 con 0x7f40b00caab0 2024-03-19T01:23:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.435+0000 7fc75affd640 1 -- 172.21.15.7:0/3651931065 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600ba020 con 0x7fc764039590 2024-03-19T01:23:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.435+0000 7fc75affd640 1 -- 172.21.15.7:0/3651931065 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7600b6700 con 0x7fc764039590 2024-03-19T01:23:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.439+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7640cd470 con 0x7fc764039590 2024-03-19T01:23:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.435+0000 7fc75affd640 1 -- 172.21.15.7:0/3651931065 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600b0040 con 0x7fc764039590 2024-03-19T01:23:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.439+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76403b010 con 0x7fc764039590 2024-03-19T01:23:35.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.439+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc764005750 con 0x7fc764039590 2024-03-19T01:23:35.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.439+0000 7fc75affd640 1 -- 172.21.15.7:0/3651931065 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc7600bd020 con 0x7fc764039590 2024-03-19T01:23:35.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.439+0000 7fc75affd640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc74c0676b0 0x7fc74c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.439+0000 7f796affd640 1 -- 172.21.15.7:0/3551601885 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f797c091a80 con 0x7f79740371f0 2024-03-19T01:23:35.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.439+0000 7fc75affd640 1 -- 172.21.15.7:0/3651931065 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc76011fe80 con 0x7fc764039590 2024-03-19T01:23:35.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.439+0000 7fc76a59c640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc74c0676b0 0x7fc74c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.443+0000 7fc76a59c640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc74c0676b0 0x7fc74c069b70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc76c0699f0 tx=0x7fc76c08cf90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.451+0000 7fc75affd640 1 -- 172.21.15.7:0/3651931065 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc7600c4030 con 0x7fc764039590 2024-03-19T01:23:35.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.487+0000 7f704ffff640 1 Processor -- start 2024-03-19T01:23:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.487+0000 7f704ffff640 1 -- start start 2024-03-19T01:23:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704ffff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 0x7f7050099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704ffff640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f705009c6d0 con 0x7f705006c6d0 2024-03-19T01:23:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704ffff640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f705006c2b0 con 0x7f705006bed0 2024-03-19T01:23:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704ffff640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f705006cb80 con 0x7f705009d850 2024-03-19T01:23:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704e7fc640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f705006c6d0 0x7f7050099660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43242/0 (socket says 172.21.15.7:43242) 2024-03-19T01:23:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704e7fc640 1 -- 172.21.15.7:0/1196940444 learned_addr learned my addr 172.21.15.7:0/1196940444 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704f7fe640 1 --2- 172.21.15.7:0/1196940444 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 0x7f7050099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704f7fe640 1 -- 172.21.15.7:0/1196940444 >> v1:172.21.15.167:6789/0 conn(0x7f705006bed0 legacy=0x7f705009d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704f7fe640 1 -- 172.21.15.7:0/1196940444 >> v1:172.21.15.7:6789/0 conn(0x7f705006c6d0 legacy=0x7f7050099660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:35.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.491+0000 7f704f7fe640 1 -- 172.21.15.7:0/1196940444 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705006ce60 con 0x7f705009d850 2024-03-19T01:23:35.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.495+0000 7f704f7fe640 1 --2- 172.21.15.7:0/1196940444 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 0x7f7050099d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7040009770 tx=0x7f704001c1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33ec82bd89457bd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704dffb640 1 -- 172.21.15.7:0/1196940444 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70400b0070 con 0x7f705009d850 2024-03-19T01:23:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704dffb640 1 -- 172.21.15.7:0/1196940444 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70400ab600 con 0x7f705009d850 2024-03-19T01:23:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704dffb640 1 -- 172.21.15.7:0/1196940444 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70400ab9d0 con 0x7f705009d850 2024-03-19T01:23:35.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704ffff640 1 -- 172.21.15.7:0/1196940444 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 msgr2=0x7f7050099d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704ffff640 1 --2- 172.21.15.7:0/1196940444 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 0x7f7050099d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7040009770 tx=0x7f704001c1f0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704ffff640 1 -- 172.21.15.7:0/1196940444 shutdown_connections 2024-03-19T01:23:35.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704ffff640 1 --2- 172.21.15.7:0/1196940444 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 0x7f7050099d70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704ffff640 1 -- 172.21.15.7:0/1196940444 >> 172.21.15.7:0/1196940444 conn(0x7f70500b86b0 msgr2=0x7f70500b8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704ffff640 1 -- 172.21.15.7:0/1196940444 shutdown_connections 2024-03-19T01:23:35.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.503+0000 7f704ffff640 1 -- 172.21.15.7:0/1196940444 wait complete. 2024-03-19T01:23:35.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 Processor -- start 2024-03-19T01:23:35.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 -- start start 2024-03-19T01:23:35.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705006bed0 0x7f70500aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f705006c6d0 0x7f70500aa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 0x7f70500aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7050151640 con 0x7f705006bed0 2024-03-19T01:23:35.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f70500b9430 con 0x7f705006c6d0 2024-03-19T01:23:35.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f70501517c0 con 0x7f705009d850 2024-03-19T01:23:35.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705006bed0 0x7f70500aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705006bed0 0x7f70500aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40172/0 (socket says 172.21.15.7:40172) 2024-03-19T01:23:35.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704effd640 1 -- 172.21.15.7:0/3323550647 learned_addr learned my addr 172.21.15.7:0/3323550647 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704e7fc640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f705006c6d0 0x7f70500aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704f7fe640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 0x7f70500aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704e7fc640 1 -- 172.21.15.7:0/3323550647 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 msgr2=0x7f70500aa5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704e7fc640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 0x7f70500aa5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704e7fc640 1 -- 172.21.15.7:0/3323550647 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705006bed0 msgr2=0x7f70500aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704e7fc640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705006bed0 0x7f70500aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704e7fc640 1 -- 172.21.15.7:0/3323550647 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705013de30 con 0x7f705006c6d0 2024-03-19T01:23:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704e7fc640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f705006c6d0 0x7f70500aa0b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f704400c3e0 tx=0x7f704400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f7037fff640 1 -- 172.21.15.7:0/3323550647 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044017070 con 0x7f705006c6d0 2024-03-19T01:23:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f7037fff640 1 -- 172.21.15.7:0/3323550647 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7044005d10 con 0x7f705006c6d0 2024-03-19T01:23:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f7037fff640 1 -- 172.21.15.7:0/3323550647 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044007310 con 0x7f705006c6d0 2024-03-19T01:23:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70500ab270 con 0x7f705006c6d0 2024-03-19T01:23:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.507+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70500abab0 con 0x7f705006c6d0 2024-03-19T01:23:35.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.511+0000 7f7037fff640 1 -- 172.21.15.7:0/3323550647 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f704401a070 con 0x7f705006c6d0 2024-03-19T01:23:35.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.511+0000 7f7037fff640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7038067880 0x7f7038069d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.511+0000 7f7037fff640 1 -- 172.21.15.7:0/3323550647 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7044084bd0 con 0x7f705006c6d0 2024-03-19T01:23:35.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.515+0000 7f704effd640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7038067880 0x7f7038069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.515+0000 7f704effd640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7038067880 0x7f7038069d40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7048002190 tx=0x7f7048009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70300052a0 con 0x7f705006c6d0 2024-03-19T01:23:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7ff8a97a2640 1 Processor -- start 2024-03-19T01:23:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7ff8a97a2640 1 -- start start 2024-03-19T01:23:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7ff8a97a2640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff8a40b31b0 0x7ff8a40b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7ff8a97a2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a40b59e0 con 0x7ff8a40b3df0 2024-03-19T01:23:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7ff8a97a2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a409c590 con 0x7ff8a4099ee0 2024-03-19T01:23:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7ff8a97a2640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7ff8a40b3590 con 0x7ff8a40b31b0 2024-03-19T01:23:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7ff8a27fc640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff8a40b3df0 0x7ff8a40997d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43244/0 (socket says 172.21.15.7:43244) 2024-03-19T01:23:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7ff8a27fc640 1 -- 172.21.15.7:0/1345087883 learned_addr learned my addr 172.21.15.7:0/1345087883 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.519+0000 7ff8a2ffd640 1 --2- 172.21.15.7:0/1345087883 >> v2:172.21.15.167:3300/0 conn(0x7ff8a40b31b0 0x7ff8a40b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.523+0000 7ff8a2ffd640 1 -- 172.21.15.7:0/1345087883 >> v1:172.21.15.167:6789/0 conn(0x7ff8a4099ee0 legacy=0x7ff8a414dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.523+0000 7ff8a2ffd640 1 -- 172.21.15.7:0/1345087883 >> v1:172.21.15.7:6789/0 conn(0x7ff8a40b3df0 legacy=0x7ff8a40997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.523+0000 7ff8a2ffd640 1 -- 172.21.15.7:0/1345087883 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a409c830 con 0x7ff8a40b31b0 2024-03-19T01:23:35.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7ff8a2ffd640 1 --2- 172.21.15.7:0/1345087883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a40b31b0 0x7ff8a40b54a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff898009fd0 tx=0x7ff89801d4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2304a8b501ae0a8a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7f7037fff640 1 -- 172.21.15.7:0/3323550647 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7044084e20 con 0x7f705006c6d0 2024-03-19T01:23:35.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7ff8a1ffb640 1 -- 172.21.15.7:0/1345087883 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8980aa650 con 0x7ff8a40b31b0 2024-03-19T01:23:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7ff8a1ffb640 1 -- 172.21.15.7:0/1345087883 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8980aa7f0 con 0x7ff8a40b31b0 2024-03-19T01:23:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7ff8a97a2640 1 -- 172.21.15.7:0/1345087883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a40b31b0 msgr2=0x7ff8a40b54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7ff8a97a2640 1 --2- 172.21.15.7:0/1345087883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a40b31b0 0x7ff8a40b54a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff898009fd0 tx=0x7ff89801d4d0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7ff8a1ffb640 1 -- 172.21.15.7:0/1345087883 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8980aabc0 con 0x7ff8a40b31b0 2024-03-19T01:23:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7ff8a97a2640 1 -- 172.21.15.7:0/1345087883 shutdown_connections 2024-03-19T01:23:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7ff8a97a2640 1 --2- 172.21.15.7:0/1345087883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a40b31b0 0x7ff8a40b54a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.531+0000 7ff8a97a2640 1 -- 172.21.15.7:0/1345087883 >> 172.21.15.7:0/1345087883 conn(0x7ff8a409a2e0 msgr2=0x7ff8a409a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 -- 172.21.15.7:0/1345087883 shutdown_connections 2024-03-19T01:23:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 -- 172.21.15.7:0/1345087883 wait complete. 2024-03-19T01:23:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 Processor -- start 2024-03-19T01:23:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 -- start start 2024-03-19T01:23:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a4099ee0 0x7ff8a40ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8a40b31b0 0x7ff8a41db7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a40b3df0 0x7ff8a41de390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a40b5fb0 con 0x7ff8a40b3df0 2024-03-19T01:23:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a409cdf0 con 0x7ff8a4099ee0 2024-03-19T01:23:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a97a2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a409add0 con 0x7ff8a40b31b0 2024-03-19T01:23:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a40b3df0 0x7ff8a41de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a40b3df0 0x7ff8a41de390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40184/0 (socket says 172.21.15.7:40184) 2024-03-19T01:23:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a37fe640 1 -- 172.21.15.7:0/986220783 learned_addr learned my addr 172.21.15.7:0/986220783 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a2ffd640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a4099ee0 0x7ff8a40ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.539+0000 7ff8a27fc640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8a40b31b0 0x7ff8a41db7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.543+0000 7ff8a2ffd640 1 -- 172.21.15.7:0/986220783 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8a40b31b0 msgr2=0x7ff8a41db7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.543+0000 7ff8a2ffd640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8a40b31b0 0x7ff8a41db7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.543+0000 7ff8a2ffd640 1 -- 172.21.15.7:0/986220783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a40b3df0 msgr2=0x7ff8a41de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.543+0000 7ff8a2ffd640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a40b3df0 0x7ff8a41de390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.543+0000 7ff8a2ffd640 1 -- 172.21.15.7:0/986220783 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a413d7f0 con 0x7ff8a4099ee0 2024-03-19T01:23:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.543+0000 7ff8a2ffd640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a4099ee0 0x7ff8a40ae880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff8980ab000 tx=0x7ff89801a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.547+0000 7ff88b7fe640 1 -- 172.21.15.7:0/986220783 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8980bf070 con 0x7ff8a4099ee0 2024-03-19T01:23:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.547+0000 7ff88b7fe640 1 -- 172.21.15.7:0/986220783 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8980a9980 con 0x7ff8a4099ee0 2024-03-19T01:23:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.547+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a41dafc0 con 0x7ff8a4099ee0 2024-03-19T01:23:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.547+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a41dbd00 con 0x7ff8a4099ee0 2024-03-19T01:23:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.547+0000 7ff88b7fe640 1 -- 172.21.15.7:0/986220783 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8980a9d50 con 0x7ff8a4099ee0 2024-03-19T01:23:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.551+0000 7ff88b7fe640 1 -- 172.21.15.7:0/986220783 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff8980c2070 con 0x7ff8a4099ee0 2024-03-19T01:23:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.551+0000 7ff88b7fe640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8900676b0 0x7ff890069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.551+0000 7ff88b7fe640 1 -- 172.21.15.7:0/986220783 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff898120160 con 0x7ff8a4099ee0 2024-03-19T01:23:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.551+0000 7ff8a27fc640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8900676b0 0x7ff890069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:35.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.551+0000 7ff8a27fc640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8900676b0 0x7ff890069b70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff894000c10 tx=0x7ff89400ad50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.555+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8840052a0 con 0x7ff8a4099ee0 2024-03-19T01:23:35.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.567+0000 7ff88b7fe640 1 -- 172.21.15.7:0/986220783 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff8980ae020 con 0x7ff8a4099ee0 2024-03-19T01:23:35.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.663+0000 7f7fed702640 1 -- 172.21.15.7:0/2941371034 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7fe80b7070 con 0x7f7fe80ae340 2024-03-19T01:23:35.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.663+0000 7f7fe4ff9640 1 -- 172.21.15.7:0/2941371034 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7fdc003810 con 0x7f7fe80ae340 2024-03-19T01:23:35.682 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2024-03-19T01:23:35.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f74540b3970 con 0x7f74540aff90 2024-03-19T01:23:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 -- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7fd4067730 msgr2=0x7f7fd4069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7fd4067730 0x7f7fd4069bf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7fe00ae000 tx=0x7f7fe00a7070 comp rx=0 tx=0).stop 2024-03-19T01:23:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 -- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fe80ae340 msgr2=0x7f7fe81323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fe80ae340 0x7f7fe81323c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00c410 tx=0x7f7fdc00c8d0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 -- 172.21.15.7:0/2941371034 shutdown_connections 2024-03-19T01:23:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7fd4067730 0x7f7fd4069bf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7fe80b52a0 0x7f7fe81f0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fe80ae340 0x7f7fe81323c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 --2- 172.21.15.7:0/2941371034 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7fe806d500 0x7f7fe8149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 -- 172.21.15.7:0/2941371034 >> 172.21.15.7:0/2941371034 conn(0x7f7fe80b5780 msgr2=0x7f7fe814d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 -- 172.21.15.7:0/2941371034 shutdown_connections 2024-03-19T01:23:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7fce7fc640 1 -- 172.21.15.7:0/2941371034 wait complete. 2024-03-19T01:23:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.687+0000 7f7449ffb640 1 -- 172.21.15.7:0/909110550 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f744c05a060 con 0x7f74540aff90 2024-03-19T01:23:35.695 INFO:teuthology.orchestra.run.smithi007.stdout:55834574854 2024-03-19T01:23:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7438067730 msgr2=0x7f7438069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7438067730 0x7f7438069bf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74440ad000 tx=0x7f74440a6000 comp rx=0 tx=0).stop 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74540aff90 msgr2=0x7f745413f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74540aff90 0x7f745413f1f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f744c00c2d0 tx=0x7f744c00c790 comp rx=0 tx=0).stop 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 shutdown_connections 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7438067730 0x7f7438069bf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7454143a90 0x7f74541403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74541446f0 0x7f7454144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 --2- 172.21.15.7:0/909110550 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74540aff90 0x7f745413f1f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 >> 172.21.15.7:0/909110550 conn(0x7f74540b5330 msgr2=0x7f74540b7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 shutdown_connections 2024-03-19T01:23:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.711+0000 7f745b6fe640 1 -- 172.21.15.7:0/909110550 wait complete. 2024-03-19T01:23:35.719 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.4 2024-03-19T01:23:35.719 DEBUG:teuthology.parallel:result is None 2024-03-19T01:23:35.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.0 2024-03-19T01:23:35.739 DEBUG:teuthology.parallel:result is None 2024-03-19T01:23:35.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.743+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f80e0005590 con 0x7f80f40c9ca0 2024-03-19T01:23:35.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.743+0000 7f80f97fa640 1 -- 172.21.15.7:0/1706201328 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f80f0016150 con 0x7f80f40c9ca0 2024-03-19T01:23:35.749 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2024-03-19T01:23:35.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ec067850 msgr2=0x7f80ec069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ec067850 0x7f80ec069d10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80fc000b70 tx=0x7f80fc08cf90 comp rx=0 tx=0).stop 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80f40c9ca0 msgr2=0x7f80f40cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80f40c9ca0 0x7f80f40cbf20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f80f00ab000 tx=0x7f80f001ad10 comp rx=0 tx=0).stop 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 shutdown_connections 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ec067850 0x7f80ec069d10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80f40c9ca0 0x7f80f40cbf20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f4038850 0x7f80f40c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 --2- 172.21.15.7:0/1706201328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80f4006d70 0x7f80f403df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 >> 172.21.15.7:0/1706201328 conn(0x7f80f4008710 msgr2=0x7f80f40ca340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 shutdown_connections 2024-03-19T01:23:35.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.747+0000 7f8101913640 1 -- 172.21.15.7:0/1706201328 wait complete. 2024-03-19T01:23:35.774 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.1 2024-03-19T01:23:35.774 DEBUG:teuthology.parallel:result is None 2024-03-19T01:23:35.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.783+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f797402b4c0 con 0x7f79740371f0 2024-03-19T01:23:35.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.783+0000 7f796affd640 1 -- 172.21.15.7:0/3551601885 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f797c16c770 con 0x7f79740371f0 2024-03-19T01:23:35.788 INFO:teuthology.orchestra.run.smithi007.stdout:55834574856 2024-03-19T01:23:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f795c067660 msgr2=0x7f795c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f795c067660 0x7f795c069b20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f797403bb60 tx=0x7f796c009060 comp rx=0 tx=0).stop 2024-03-19T01:23:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 msgr2=0x7f79740cd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f79740cd2a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f797c000b70 tx=0x7f797c094b40 comp rx=0 tx=0).stop 2024-03-19T01:23:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 shutdown_connections 2024-03-19T01:23:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f795c067660 0x7f795c069b20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79740cd910 0x7f79740cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79740c4850 0x7f79740b6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 --2- 172.21.15.7:0/3551601885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79740371f0 0x7f79740cd2a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 >> 172.21.15.7:0/3551601885 conn(0x7f797403a890 msgr2=0x7f7974038990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 shutdown_connections 2024-03-19T01:23:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.787+0000 7f797b59e640 1 -- 172.21.15.7:0/3551601885 wait complete. 2024-03-19T01:23:35.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f40a0005590 con 0x7f40b00caab0 2024-03-19T01:23:35.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40ae7fc640 1 -- 172.21.15.7:0/2766020310 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f40b806c340 con 0x7f40b00caab0 2024-03-19T01:23:35.796 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2024-03-19T01:23:35.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f409c067790 msgr2=0x7f409c069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f409c067790 0x7f409c069c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009630 tx=0x7f40a8006ea0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40b00caab0 msgr2=0x7f40b0175000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40b00caab0 0x7f40b0175000 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f40b804f860 tx=0x7f40b8069570 comp rx=0 tx=0).stop 2024-03-19T01:23:35.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 shutdown_connections 2024-03-19T01:23:35.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f409c067790 0x7f409c069c50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40b00caab0 0x7f40b0175000 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40b0038850 0x7f40b00afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 --2- 172.21.15.7:0/2766020310 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40b0006d70 0x7f40b00cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 >> 172.21.15.7:0/2766020310 conn(0x7f40b0008710 msgr2=0x7f40b0004e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 shutdown_connections 2024-03-19T01:23:35.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.791+0000 7f40be8af640 1 -- 172.21.15.7:0/2766020310 wait complete. 2024-03-19T01:23:35.808 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.7 2024-03-19T01:23:35.809 DEBUG:teuthology.parallel:result is None 2024-03-19T01:23:35.815 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.5 2024-03-19T01:23:35.815 DEBUG:teuthology.parallel:result is None 2024-03-19T01:23:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.823+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc76403cb60 con 0x7fc764039590 2024-03-19T01:23:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.823+0000 7fc75affd640 1 -- 172.21.15.7:0/3651931065 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc760016170 con 0x7fc764039590 2024-03-19T01:23:35.830 INFO:teuthology.orchestra.run.smithi007.stdout:55834574856 2024-03-19T01:23:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc74c0676b0 msgr2=0x7fc74c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc74c0676b0 0x7fc74c069b70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc76c0699f0 tx=0x7fc76c08cf90 comp rx=0 tx=0).stop 2024-03-19T01:23:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc764039590 msgr2=0x7fc7640c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc764039590 0x7fc7640c9c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc7600ab000 tx=0x7fc76001aca0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 shutdown_connections 2024-03-19T01:23:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc74c0676b0 0x7fc74c069b70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7640d1130 0x7fc7640cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc764039590 0x7fc7640c9c90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 --2- 172.21.15.7:0/3651931065 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc764007f60 0x7fc7640ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 >> 172.21.15.7:0/3651931065 conn(0x7fc76403c6e0 msgr2=0x7fc764006390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 shutdown_connections 2024-03-19T01:23:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.827+0000 7fc770b1a640 1 -- 172.21.15.7:0/3651931065 wait complete. 2024-03-19T01:23:35.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.6 2024-03-19T01:23:35.853 DEBUG:teuthology.parallel:result is None 2024-03-19T01:23:35.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f7030005590 con 0x7f705006c6d0 2024-03-19T01:23:35.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f7037fff640 1 -- 172.21.15.7:0/3323550647 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f70500ab270 con 0x7f705006c6d0 2024-03-19T01:23:35.860 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2024-03-19T01:23:35.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7038067880 msgr2=0x7f7038069d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7038067880 0x7f7038069d40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7048002190 tx=0x7f7048009040 comp rx=0 tx=0).stop 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f705006c6d0 msgr2=0x7f70500aa0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f705006c6d0 0x7f70500aa0b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f704400c3e0 tx=0x7f704400c8a0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 shutdown_connections 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7038067880 0x7f7038069d40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f705009d850 0x7f70500aa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f705006c6d0 0x7f70500aa0b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 --2- 172.21.15.7:0/3323550647 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f705006bed0 0x7f70500aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 >> 172.21.15.7:0/3323550647 conn(0x7f70500b86b0 msgr2=0x7f70500b6490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 shutdown_connections 2024-03-19T01:23:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.855+0000 7f704ffff640 1 -- 172.21.15.7:0/3323550647 wait complete. 2024-03-19T01:23:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.867+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff884005860 con 0x7ff8a4099ee0 2024-03-19T01:23:35.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.867+0000 7ff88b7fe640 1 -- 172.21.15.7:0/986220783 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff8980171e0 con 0x7ff8a4099ee0 2024-03-19T01:23:35.874 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2024-03-19T01:23:35.877 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.3 2024-03-19T01:23:35.877 DEBUG:teuthology.parallel:result is None 2024-03-19T01:23:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8900676b0 msgr2=0x7ff890069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8900676b0 0x7ff890069b70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff894000c10 tx=0x7ff89400ad50 comp rx=0 tx=0).stop 2024-03-19T01:23:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a4099ee0 msgr2=0x7ff8a40ae880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a4099ee0 0x7ff8a40ae880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff8980ab000 tx=0x7ff89801a270 comp rx=0 tx=0).stop 2024-03-19T01:23:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 shutdown_connections 2024-03-19T01:23:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8900676b0 0x7ff890069b70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8a40b31b0 0x7ff8a41db7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a40b3df0 0x7ff8a41de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 --2- 172.21.15.7:0/986220783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8a4099ee0 0x7ff8a40ae880 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 >> 172.21.15.7:0/986220783 conn(0x7ff8a409a2e0 msgr2=0x7ff8a40b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 shutdown_connections 2024-03-19T01:23:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:35.871+0000 7ff8a97a2640 1 -- 172.21.15.7:0/986220783 wait complete. 2024-03-19T01:23:35.891 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.2 2024-03-19T01:23:35.891 DEBUG:teuthology.parallel:result is None 2024-03-19T01:23:35.891 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-19T01:23:35.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:23:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041c7c8640 1 Processor -- start 2024-03-19T01:23:36.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041c7c8640 1 -- start start 2024-03-19T01:23:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041c7c8640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f04140add50 0x7f0414147830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041c7c8640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414147d70 0x7f0414151d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041c7c8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04140aead0 con 0x7f0414074390 2024-03-19T01:23:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041c7c8640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f04141481c0 con 0x7f04140add50 2024-03-19T01:23:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041c7c8640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0414074770 con 0x7f0414147d70 2024-03-19T01:23:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041a564640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0414074390 0x7f0414141430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43254/0 (socket says 172.21.15.7:43254) 2024-03-19T01:23:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041a564640 1 -- 172.21.15.7:0/2750231137 learned_addr learned my addr 172.21.15.7:0/2750231137 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f041ad65640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414147d70 0x7f0414151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f0419d63640 1 --2- 172.21.15.7:0/2750231137 >> v2:172.21.15.167:3300/0 conn(0x7f04140add50 0x7f0414147830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793441480 0 0) 0x7f04140aead0 con 0x7f0414074390 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.043+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0400003530 con 0x7f0414074390 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1151513689 0 0) 0x7f0400003530 con 0x7f0414074390 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414147d70 msgr2=0x7f0414151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414147d70 0x7f0414151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 >> v2:172.21.15.167:3300/0 conn(0x7f04140add50 msgr2=0x7f0414147830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 --2- 172.21.15.7:0/2750231137 >> v2:172.21.15.167:3300/0 conn(0x7f04140add50 0x7f0414147830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04140af050 con 0x7f0414074390 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f04100016a0 con 0x7f0414074390 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 >> v1:172.21.15.7:6789/0 conn(0x7f0414074390 legacy=0x7f0414141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04000039c0 0x7f0400003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f0400004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414147d70 0x7f04000051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f04141481c0 con 0x7f0400004810 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0414074770 con 0x7f04000039c0 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f040000ba50 con 0x7f0414147d70 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419d63640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f0400004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f041a564640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04000039c0 0x7f0400003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f041ad65640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414147d70 0x7f04000051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419d63640 1 -- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414147d70 msgr2=0x7f04000051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419d63640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414147d70 0x7f04000051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419d63640 1 -- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04000039c0 msgr2=0x7f0400003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419d63640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04000039c0 0x7f0400003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419d63640 1 -- 172.21.15.7:0/2750231137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0400003530 con 0x7f0400004810 2024-03-19T01:23:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.047+0000 7f0419d63640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f0400004c90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0404009ab0 tx=0x7f040401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce31f42b2a65f96c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04040b2070 con 0x7f0400004810 2024-03-19T01:23:36.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04040a5530 con 0x7f0400004810 2024-03-19T01:23:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f0419562640 1 -- 172.21.15.7:0/2750231137 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04040096b0 con 0x7f0400004810 2024-03-19T01:23:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- 172.21.15.7:0/2750231137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 msgr2=0x7f0400004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 --2- 172.21.15.7:0/2750231137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f0400004c90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0404009ab0 tx=0x7f040401ca90 comp rx=0 tx=0).stop 2024-03-19T01:23:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041ad65640 1 -- 172.21.15.7:0/2750231137 reap_dead start 2024-03-19T01:23:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- 172.21.15.7:0/2750231137 shutdown_connections 2024-03-19T01:23:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- 172.21.15.7:0/2750231137 >> 172.21.15.7:0/2750231137 conn(0x7f04140b5bf0 msgr2=0x7f04140b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- 172.21.15.7:0/2750231137 shutdown_connections 2024-03-19T01:23:36.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- 172.21.15.7:0/2750231137 wait complete. 2024-03-19T01:23:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 Processor -- start 2024-03-19T01:23:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- start start 2024-03-19T01:23:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f041413e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f041413d890 0x7f0414142980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414142ec0 0x7f04140b1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f04140ae360 con 0x7f0400004810 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f04140b5860 con 0x7f041413d890 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f04140ae4e0 con 0x7f0414142ec0 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041a564640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f041413e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041a564640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f041413e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40198/0 (socket says 172.21.15.7:40198) 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041a564640 1 -- 172.21.15.7:0/106553280 learned_addr learned my addr 172.21.15.7:0/106553280 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041ad65640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414142ec0 0x7f04140b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f0419d63640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f041413d890 0x7f0414142980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041a564640 1 -- 172.21.15.7:0/106553280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414142ec0 msgr2=0x7f04140b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041a564640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414142ec0 0x7f04140b1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041a564640 1 -- 172.21.15.7:0/106553280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f041413d890 msgr2=0x7f0414142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041a564640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f041413d890 0x7f0414142980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041a564640 1 -- 172.21.15.7:0/106553280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f041413d1c0 con 0x7f0400004810 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f0419d63640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f041413d890 0x7f0414142980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041a564640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f041413e2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f041000e7e0 tx=0x7f041000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f040affd640 1 -- 172.21.15.7:0/106553280 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041000c6c0 con 0x7f0400004810 2024-03-19T01:23:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f040affd640 1 -- 172.21.15.7:0/106553280 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0410018070 con 0x7f0400004810 2024-03-19T01:23:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f040affd640 1 -- 172.21.15.7:0/106553280 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041000cac0 con 0x7f0400004810 2024-03-19T01:23:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04141421a0 con 0x7f0400004810 2024-03-19T01:23:36.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.051+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f041413dd10 con 0x7f0400004810 2024-03-19T01:23:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.055+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04141432a0 con 0x7f0400004810 2024-03-19T01:23:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.055+0000 7f040affd640 1 -- 172.21.15.7:0/106553280 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f041000cc60 con 0x7f0400004810 2024-03-19T01:23:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.055+0000 7f040affd640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03f8067660 0x7f03f8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.055+0000 7f0419d63640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03f8067660 0x7f03f8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.055+0000 7f040affd640 1 -- 172.21.15.7:0/106553280 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f04100855f0 con 0x7f0400004810 2024-03-19T01:23:36.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.055+0000 7f0419d63640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03f8067660 0x7f03f8069b20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f04040af000 tx=0x7f04040046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.059+0000 7f040affd640 1 -- 172.21.15.7:0/106553280 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0410056270 con 0x7f0400004810 2024-03-19T01:23:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.211+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0414075460 con 0x7f03f8067660 2024-03-19T01:23:36.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.227+0000 7f040affd640 1 -- 172.21.15.7:0/106553280 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f0414075460 con 0x7f03f8067660 2024-03-19T01:23:36.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:36.235 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:23:36.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03f8067660 msgr2=0x7f03f8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03f8067660 0x7f03f8069b20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f04040af000 tx=0x7f04040046d0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 msgr2=0x7f041413e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f041413e2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f041000e7e0 tx=0x7f041000eca0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 shutdown_connections 2024-03-19T01:23:36.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03f8067660 0x7f03f8069b20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0414142ec0 0x7f04140b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f041413d890 0x7f0414142980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 --2- 172.21.15.7:0/106553280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0400004810 0x7f041413e2f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 >> 172.21.15.7:0/106553280 conn(0x7f04140b5bf0 msgr2=0x7f0414141200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:36.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 shutdown_connections 2024-03-19T01:23:36.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.231+0000 7f041c7c8640 1 -- 172.21.15.7:0/106553280 wait complete. 2024-03-19T01:23:36.251 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-19T01:23:34.892348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235756,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738964,"statfs":{"total":773094113280,"available":772852699136,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002750"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333975+0000","last_change":"2024-03-19T01:23:27.551581+0000","last_active":"2024-03-19T01:23:29.333975+0000","last_peered":"2024-03-19T01:23:29.333975+0000","last_clean":"2024-03-19T01:23:29.333975+0000","last_became_active":"2024-03-19T01:23:27.550909+0000","last_became_peered":"2024-03-19T01:23:27.550909+0000","last_unstale":"2024-03-19T01:23:29.333975+0000","last_undegraded":"2024-03-19T01:23:29.333975+0000","last_fullsized":"2024-03-19T01:23:29.333975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:14.092827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498885+0000","last_change":"2024-03-19T01:23:27.512582+0000","last_active":"2024-03-19T01:23:28.498885+0000","last_peered":"2024-03-19T01:23:28.498885+0000","last_clean":"2024-03-19T01:23:28.498885+0000","last_became_active":"2024-03-19T01:23:27.512252+0000","last_became_peered":"2024-03-19T01:23:27.512252+0000","last_unstale":"2024-03-19T01:23:28.498885+0000","last_undegraded":"2024-03-19T01:23:28.498885+0000","last_fullsized":"2024-03-19T01:23:28.498885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331920+0000","last_change":"2024-03-19T01:23:27.512676+0000","last_active":"2024-03-19T01:23:29.331920+0000","last_peered":"2024-03-19T01:23:29.331920+0000","last_clean":"2024-03-19T01:23:29.331920+0000","last_became_active":"2024-03-19T01:23:27.512311+0000","last_became_peered":"2024-03-19T01:23:27.512311+0000","last_unstale":"2024-03-19T01:23:29.331920+0000","last_undegraded":"2024-03-19T01:23:29.331920+0000","last_fullsized":"2024-03-19T01:23:29.331920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498730+0000","last_change":"2024-03-19T01:23:27.538857+0000","last_active":"2024-03-19T01:23:28.498730+0000","last_peered":"2024-03-19T01:23:28.498730+0000","last_clean":"2024-03-19T01:23:28.498730+0000","last_became_active":"2024-03-19T01:23:27.536808+0000","last_became_peered":"2024-03-19T01:23:27.536808+0000","last_unstale":"2024-03-19T01:23:28.498730+0000","last_undegraded":"2024-03-19T01:23:28.498730+0000","last_fullsized":"2024-03-19T01:23:28.498730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332035+0000","last_change":"2024-03-19T01:23:27.543517+0000","last_active":"2024-03-19T01:23:29.332035+0000","last_peered":"2024-03-19T01:23:29.332035+0000","last_clean":"2024-03-19T01:23:29.332035+0000","last_became_active":"2024-03-19T01:23:27.542861+0000","last_became_peered":"2024-03-19T01:23:27.542861+0000","last_unstale":"2024-03-19T01:23:29.332035+0000","last_undegraded":"2024-03-19T01:23:29.332035+0000","last_fullsized":"2024-03-19T01:23:29.332035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498639+0000","last_change":"2024-03-19T01:23:27.549333+0000","last_active":"2024-03-19T01:23:28.498639+0000","last_peered":"2024-03-19T01:23:28.498639+0000","last_clean":"2024-03-19T01:23:28.498639+0000","last_became_active":"2024-03-19T01:23:27.548338+0000","last_became_peered":"2024-03-19T01:23:27.548338+0000","last_unstale":"2024-03-19T01:23:28.498639+0000","last_undegraded":"2024-03-19T01:23:28.498639+0000","last_fullsized":"2024-03-19T01:23:28.498639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331558+0000","last_change":"2024-03-19T01:23:27.536206+0000","last_active":"2024-03-19T01:23:29.331558+0000","last_peered":"2024-03-19T01:23:29.331558+0000","last_clean":"2024-03-19T01:23:29.331558+0000","last_became_active":"2024-03-19T01:23:27.535402+0000","last_became_peered":"2024-03-19T01:23:27.535402+0000","last_unstale":"2024-03-19T01:23:29.331558+0000","last_undegraded":"2024-03-19T01:23:29.331558+0000","last_fullsized":"2024-03-19T01:23:29.331558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569176+0000","last_change":"2024-03-19T01:23:27.548444+0000","last_active":"2024-03-19T01:23:28.569176+0000","last_peered":"2024-03-19T01:23:28.569176+0000","last_clean":"2024-03-19T01:23:28.569176+0000","last_became_active":"2024-03-19T01:23:27.548160+0000","last_became_peered":"2024-03-19T01:23:27.548160+0000","last_unstale":"2024-03-19T01:23:28.569176+0000","last_undegraded":"2024-03-19T01:23:28.569176+0000","last_fullsized":"2024-03-19T01:23:28.569176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.334001+0000","last_change":"2024-03-19T01:23:27.558139+0000","last_active":"2024-03-19T01:23:29.334001+0000","last_peered":"2024-03-19T01:23:29.334001+0000","last_clean":"2024-03-19T01:23:29.334001+0000","last_became_active":"2024-03-19T01:23:27.557564+0000","last_became_peered":"2024-03-19T01:23:27.557564+0000","last_unstale":"2024-03-19T01:23:29.334001+0000","last_undegraded":"2024-03-19T01:23:29.334001+0000","last_fullsized":"2024-03-19T01:23:29.334001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:31:46.209018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333331+0000","last_change":"2024-03-19T01:23:27.549847+0000","last_active":"2024-03-19T01:23:29.333331+0000","last_peered":"2024-03-19T01:23:29.333331+0000","last_clean":"2024-03-19T01:23:29.333331+0000","last_became_active":"2024-03-19T01:23:27.548601+0000","last_became_peered":"2024-03-19T01:23:27.548601+0000","last_unstale":"2024-03-19T01:23:29.333331+0000","last_undegraded":"2024-03-19T01:23:29.333331+0000","last_fullsized":"2024-03-19T01:23:29.333331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333016+0000","last_change":"2024-03-19T01:23:27.558581+0000","last_active":"2024-03-19T01:23:29.333016+0000","last_peered":"2024-03-19T01:23:29.333016+0000","last_clean":"2024-03-19T01:23:29.333016+0000","last_became_active":"2024-03-19T01:23:27.558044+0000","last_became_peered":"2024-03-19T01:23:27.558044+0000","last_unstale":"2024-03-19T01:23:29.333016+0000","last_undegraded":"2024-03-19T01:23:29.333016+0000","last_fullsized":"2024-03-19T01:23:29.333016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831030+0000","last_change":"2024-03-19T01:23:27.543968+0000","last_active":"2024-03-19T01:23:28.831030+0000","last_peered":"2024-03-19T01:23:28.831030+0000","last_clean":"2024-03-19T01:23:28.831030+0000","last_became_active":"2024-03-19T01:23:27.543453+0000","last_became_peered":"2024-03-19T01:23:27.543453+0000","last_unstale":"2024-03-19T01:23:28.831030+0000","last_undegraded":"2024-03-19T01:23:28.831030+0000","last_fullsized":"2024-03-19T01:23:28.831030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569582+0000","last_change":"2024-03-19T01:23:27.541445+0000","last_active":"2024-03-19T01:23:28.569582+0000","last_peered":"2024-03-19T01:23:28.569582+0000","last_clean":"2024-03-19T01:23:28.569582+0000","last_became_active":"2024-03-19T01:23:27.541162+0000","last_became_peered":"2024-03-19T01:23:27.541162+0000","last_unstale":"2024-03-19T01:23:28.569582+0000","last_undegraded":"2024-03-19T01:23:28.569582+0000","last_fullsized":"2024-03-19T01:23:28.569582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331530+0000","last_change":"2024-03-19T01:23:27.515384+0000","last_active":"2024-03-19T01:23:29.331530+0000","last_peered":"2024-03-19T01:23:29.331530+0000","last_clean":"2024-03-19T01:23:29.331530+0000","last_became_active":"2024-03-19T01:23:27.514935+0000","last_became_peered":"2024-03-19T01:23:27.514935+0000","last_unstale":"2024-03-19T01:23:29.331530+0000","last_undegraded":"2024-03-19T01:23:29.331530+0000","last_fullsized":"2024-03-19T01:23:29.331530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:52.938759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497668+0000","last_change":"2024-03-19T01:23:27.509553+0000","last_active":"2024-03-19T01:23:28.497668+0000","last_peered":"2024-03-19T01:23:28.497668+0000","last_clean":"2024-03-19T01:23:28.497668+0000","last_became_active":"2024-03-19T01:23:27.509178+0000","last_became_peered":"2024-03-19T01:23:27.509178+0000","last_unstale":"2024-03-19T01:23:28.497668+0000","last_undegraded":"2024-03-19T01:23:28.497668+0000","last_fullsized":"2024-03-19T01:23:28.497668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498214+0000","last_change":"2024-03-19T01:23:27.547056+0000","last_active":"2024-03-19T01:23:28.498214+0000","last_peered":"2024-03-19T01:23:28.498214+0000","last_clean":"2024-03-19T01:23:28.498214+0000","last_became_active":"2024-03-19T01:23:27.538339+0000","last_became_peered":"2024-03-19T01:23:27.538339+0000","last_unstale":"2024-03-19T01:23:28.498214+0000","last_undegraded":"2024-03-19T01:23:28.498214+0000","last_fullsized":"2024-03-19T01:23:28.498214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.500126+0000","last_change":"2024-03-19T01:23:27.562551+0000","last_active":"2024-03-19T01:23:28.500126+0000","last_peered":"2024-03-19T01:23:28.500126+0000","last_clean":"2024-03-19T01:23:28.500126+0000","last_became_active":"2024-03-19T01:23:27.561996+0000","last_became_peered":"2024-03-19T01:23:27.561996+0000","last_unstale":"2024-03-19T01:23:28.500126+0000","last_undegraded":"2024-03-19T01:23:28.500126+0000","last_fullsized":"2024-03-19T01:23:28.500126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:27.331497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572895+0000","last_change":"2024-03-19T01:23:27.541038+0000","last_active":"2024-03-19T01:23:28.572895+0000","last_peered":"2024-03-19T01:23:28.572895+0000","last_clean":"2024-03-19T01:23:28.572895+0000","last_became_active":"2024-03-19T01:23:27.540751+0000","last_became_peered":"2024-03-19T01:23:27.540751+0000","last_unstale":"2024-03-19T01:23:28.572895+0000","last_undegraded":"2024-03-19T01:23:28.572895+0000","last_fullsized":"2024-03-19T01:23:28.572895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333787+0000","last_change":"2024-03-19T01:23:27.552053+0000","last_active":"2024-03-19T01:23:29.333787+0000","last_peered":"2024-03-19T01:23:29.333787+0000","last_clean":"2024-03-19T01:23:29.333787+0000","last_became_active":"2024-03-19T01:23:27.551405+0000","last_became_peered":"2024-03-19T01:23:27.551405+0000","last_unstale":"2024-03-19T01:23:29.333787+0000","last_undegraded":"2024-03-19T01:23:29.333787+0000","last_fullsized":"2024-03-19T01:23:29.333787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:27.331497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332923+0000","last_change":"2024-03-19T01:23:27.548149+0000","last_active":"2024-03-19T01:23:29.332923+0000","last_peered":"2024-03-19T01:23:29.332923+0000","last_clean":"2024-03-19T01:23:29.332923+0000","last_became_active":"2024-03-19T01:23:27.542105+0000","last_became_peered":"2024-03-19T01:23:27.542105+0000","last_unstale":"2024-03-19T01:23:29.332923+0000","last_undegraded":"2024-03-19T01:23:29.332923+0000","last_fullsized":"2024-03-19T01:23:29.332923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332234+0000","last_change":"2024-03-19T01:23:27.521814+0000","last_active":"2024-03-19T01:23:29.332234+0000","last_peered":"2024-03-19T01:23:29.332234+0000","last_clean":"2024-03-19T01:23:29.332234+0000","last_became_active":"2024-03-19T01:23:27.519639+0000","last_became_peered":"2024-03-19T01:23:27.519639+0000","last_unstale":"2024-03-19T01:23:29.332234+0000","last_undegraded":"2024-03-19T01:23:29.332234+0000","last_fullsized":"2024-03-19T01:23:29.332234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499387+0000","last_change":"2024-03-19T01:23:27.558248+0000","last_active":"2024-03-19T01:23:28.499387+0000","last_peered":"2024-03-19T01:23:28.499387+0000","last_clean":"2024-03-19T01:23:28.499387+0000","last_became_active":"2024-03-19T01:23:27.557663+0000","last_became_peered":"2024-03-19T01:23:27.557663+0000","last_unstale":"2024-03-19T01:23:28.499387+0000","last_undegraded":"2024-03-19T01:23:28.499387+0000","last_fullsized":"2024-03-19T01:23:28.499387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.568928+0000","last_change":"2024-03-19T01:23:27.532119+0000","last_active":"2024-03-19T01:23:28.568928+0000","last_peered":"2024-03-19T01:23:28.568928+0000","last_clean":"2024-03-19T01:23:28.568928+0000","last_became_active":"2024-03-19T01:23:27.531846+0000","last_became_peered":"2024-03-19T01:23:27.531846+0000","last_unstale":"2024-03-19T01:23:28.568928+0000","last_undegraded":"2024-03-19T01:23:28.568928+0000","last_fullsized":"2024-03-19T01:23:28.568928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498373+0000","last_change":"2024-03-19T01:23:27.546292+0000","last_active":"2024-03-19T01:23:28.498373+0000","last_peered":"2024-03-19T01:23:28.498373+0000","last_clean":"2024-03-19T01:23:28.498373+0000","last_became_active":"2024-03-19T01:23:27.537369+0000","last_became_peered":"2024-03-19T01:23:27.537369+0000","last_unstale":"2024-03-19T01:23:28.498373+0000","last_undegraded":"2024-03-19T01:23:28.498373+0000","last_fullsized":"2024-03-19T01:23:28.498373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569601+0000","last_change":"2024-03-19T01:23:27.522448+0000","last_active":"2024-03-19T01:23:28.569601+0000","last_peered":"2024-03-19T01:23:28.569601+0000","last_clean":"2024-03-19T01:23:28.569601+0000","last_became_active":"2024-03-19T01:23:27.522201+0000","last_became_peered":"2024-03-19T01:23:27.522201+0000","last_unstale":"2024-03-19T01:23:28.569601+0000","last_undegraded":"2024-03-19T01:23:28.569601+0000","last_fullsized":"2024-03-19T01:23:28.569601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333402+0000","last_change":"2024-03-19T01:23:27.551178+0000","last_active":"2024-03-19T01:23:29.333402+0000","last_peered":"2024-03-19T01:23:29.333402+0000","last_clean":"2024-03-19T01:23:29.333402+0000","last_became_active":"2024-03-19T01:23:27.550571+0000","last_became_peered":"2024-03-19T01:23:27.550571+0000","last_unstale":"2024-03-19T01:23:29.333402+0000","last_undegraded":"2024-03-19T01:23:29.333402+0000","last_fullsized":"2024-03-19T01:23:29.333402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497894+0000","last_change":"2024-03-19T01:23:27.509194+0000","last_active":"2024-03-19T01:23:28.497894+0000","last_peered":"2024-03-19T01:23:28.497894+0000","last_clean":"2024-03-19T01:23:28.497894+0000","last_became_active":"2024-03-19T01:23:27.508796+0000","last_became_peered":"2024-03-19T01:23:27.508796+0000","last_unstale":"2024-03-19T01:23:28.497894+0000","last_undegraded":"2024-03-19T01:23:28.497894+0000","last_fullsized":"2024-03-19T01:23:28.497894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569258+0000","last_change":"2024-03-19T01:23:27.522380+0000","last_active":"2024-03-19T01:23:28.569258+0000","last_peered":"2024-03-19T01:23:28.569258+0000","last_clean":"2024-03-19T01:23:28.569258+0000","last_became_active":"2024-03-19T01:23:27.521968+0000","last_became_peered":"2024-03-19T01:23:27.521968+0000","last_unstale":"2024-03-19T01:23:28.569258+0000","last_undegraded":"2024-03-19T01:23:28.569258+0000","last_fullsized":"2024-03-19T01:23:28.569258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332566+0000","last_change":"2024-03-19T01:23:27.516056+0000","last_active":"2024-03-19T01:23:29.332566+0000","last_peered":"2024-03-19T01:23:29.332566+0000","last_clean":"2024-03-19T01:23:29.332566+0000","last_became_active":"2024-03-19T01:23:27.515651+0000","last_became_peered":"2024-03-19T01:23:27.515651+0000","last_unstale":"2024-03-19T01:23:29.332566+0000","last_undegraded":"2024-03-19T01:23:29.332566+0000","last_fullsized":"2024-03-19T01:23:29.332566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499052+0000","last_change":"2024-03-19T01:23:27.547751+0000","last_active":"2024-03-19T01:23:28.499052+0000","last_peered":"2024-03-19T01:23:28.499052+0000","last_clean":"2024-03-19T01:23:28.499052+0000","last_became_active":"2024-03-19T01:23:27.530027+0000","last_became_peered":"2024-03-19T01:23:27.530027+0000","last_unstale":"2024-03-19T01:23:28.499052+0000","last_undegraded":"2024-03-19T01:23:28.499052+0000","last_fullsized":"2024-03-19T01:23:28.499052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572767+0000","last_change":"2024-03-19T01:23:27.538411+0000","last_active":"2024-03-19T01:23:28.572767+0000","last_peered":"2024-03-19T01:23:28.572767+0000","last_clean":"2024-03-19T01:23:28.572767+0000","last_became_active":"2024-03-19T01:23:27.537893+0000","last_became_peered":"2024-03-19T01:23:27.537893+0000","last_unstale":"2024-03-19T01:23:28.572767+0000","last_undegraded":"2024-03-19T01:23:28.572767+0000","last_fullsized":"2024-03-19T01:23:28.572767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331763+0000","last_change":"2024-03-19T01:23:27.539735+0000","last_active":"2024-03-19T01:23:29.331763+0000","last_peered":"2024-03-19T01:23:29.331763+0000","last_clean":"2024-03-19T01:23:29.331763+0000","last_became_active":"2024-03-19T01:23:27.539022+0000","last_became_peered":"2024-03-19T01:23:27.539022+0000","last_unstale":"2024-03-19T01:23:29.331763+0000","last_undegraded":"2024-03-19T01:23:29.331763+0000","last_fullsized":"2024-03-19T01:23:29.331763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830822+0000","last_change":"2024-03-19T01:23:27.503767+0000","last_active":"2024-03-19T01:23:28.830822+0000","last_peered":"2024-03-19T01:23:28.830822+0000","last_clean":"2024-03-19T01:23:28.830822+0000","last_became_active":"2024-03-19T01:23:27.503030+0000","last_became_peered":"2024-03-19T01:23:27.503030+0000","last_unstale":"2024-03-19T01:23:28.830822+0000","last_undegraded":"2024-03-19T01:23:28.830822+0000","last_fullsized":"2024-03-19T01:23:28.830822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:08.835543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569515+0000","last_change":"2024-03-19T01:23:26.532832+0000","last_active":"2024-03-19T01:23:28.569515+0000","last_peered":"2024-03-19T01:23:28.569515+0000","last_clean":"2024-03-19T01:23:28.569515+0000","last_became_active":"2024-03-19T01:23:26.532516+0000","last_became_peered":"2024-03-19T01:23:26.532516+0000","last_unstale":"2024-03-19T01:23:28.569515+0000","last_undegraded":"2024-03-19T01:23:28.569515+0000","last_fullsized":"2024-03-19T01:23:28.569515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332652+0000","last_change":"2024-03-19T01:23:27.522605+0000","last_active":"2024-03-19T01:23:29.332652+0000","last_peered":"2024-03-19T01:23:29.332652+0000","last_clean":"2024-03-19T01:23:29.332652+0000","last_became_active":"2024-03-19T01:23:27.522344+0000","last_became_peered":"2024-03-19T01:23:27.522344+0000","last_unstale":"2024-03-19T01:23:29.332652+0000","last_undegraded":"2024-03-19T01:23:29.332652+0000","last_fullsized":"2024-03-19T01:23:29.332652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332682+0000","last_change":"2024-03-19T01:23:26.535891+0000","last_active":"2024-03-19T01:23:29.332682+0000","last_peered":"2024-03-19T01:23:29.332682+0000","last_clean":"2024-03-19T01:23:29.332682+0000","last_became_active":"2024-03-19T01:23:26.535218+0000","last_became_peered":"2024-03-19T01:23:26.535218+0000","last_unstale":"2024-03-19T01:23:29.332682+0000","last_undegraded":"2024-03-19T01:23:29.332682+0000","last_fullsized":"2024-03-19T01:23:29.332682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.568845+0000","last_change":"2024-03-19T01:23:26.536647+0000","last_active":"2024-03-19T01:23:28.568845+0000","last_peered":"2024-03-19T01:23:28.568845+0000","last_clean":"2024-03-19T01:23:28.568845+0000","last_became_active":"2024-03-19T01:23:26.535919+0000","last_became_peered":"2024-03-19T01:23:26.535919+0000","last_unstale":"2024-03-19T01:23:28.568845+0000","last_undegraded":"2024-03-19T01:23:28.568845+0000","last_fullsized":"2024-03-19T01:23:28.568845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569044+0000","last_change":"2024-03-19T01:23:27.517889+0000","last_active":"2024-03-19T01:23:28.569044+0000","last_peered":"2024-03-19T01:23:28.569044+0000","last_clean":"2024-03-19T01:23:28.569044+0000","last_became_active":"2024-03-19T01:23:27.515221+0000","last_became_peered":"2024-03-19T01:23:27.515221+0000","last_unstale":"2024-03-19T01:23:28.569044+0000","last_undegraded":"2024-03-19T01:23:28.569044+0000","last_fullsized":"2024-03-19T01:23:28.569044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499348+0000","last_change":"2024-03-19T01:23:26.518817+0000","last_active":"2024-03-19T01:23:28.499348+0000","last_peered":"2024-03-19T01:23:28.499348+0000","last_clean":"2024-03-19T01:23:28.499348+0000","last_became_active":"2024-03-19T01:23:26.518008+0000","last_became_peered":"2024-03-19T01:23:26.518008+0000","last_unstale":"2024-03-19T01:23:28.499348+0000","last_undegraded":"2024-03-19T01:23:28.499348+0000","last_fullsized":"2024-03-19T01:23:28.499348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498152+0000","last_change":"2024-03-19T01:23:27.514550+0000","last_active":"2024-03-19T01:23:28.498152+0000","last_peered":"2024-03-19T01:23:28.498152+0000","last_clean":"2024-03-19T01:23:28.498152+0000","last_became_active":"2024-03-19T01:23:27.512624+0000","last_became_peered":"2024-03-19T01:23:27.512624+0000","last_unstale":"2024-03-19T01:23:28.498152+0000","last_undegraded":"2024-03-19T01:23:28.498152+0000","last_fullsized":"2024-03-19T01:23:28.498152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332747+0000","last_change":"2024-03-19T01:23:26.536461+0000","last_active":"2024-03-19T01:23:29.332747+0000","last_peered":"2024-03-19T01:23:29.332747+0000","last_clean":"2024-03-19T01:23:29.332747+0000","last_became_active":"2024-03-19T01:23:26.535707+0000","last_became_peered":"2024-03-19T01:23:26.535707+0000","last_unstale":"2024-03-19T01:23:29.332747+0000","last_undegraded":"2024-03-19T01:23:29.332747+0000","last_fullsized":"2024-03-19T01:23:29.332747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499599+0000","last_change":"2024-03-19T01:23:27.561680+0000","last_active":"2024-03-19T01:23:28.499599+0000","last_peered":"2024-03-19T01:23:28.499599+0000","last_clean":"2024-03-19T01:23:28.499599+0000","last_became_active":"2024-03-19T01:23:27.559667+0000","last_became_peered":"2024-03-19T01:23:27.559667+0000","last_unstale":"2024-03-19T01:23:28.499599+0000","last_undegraded":"2024-03-19T01:23:28.499599+0000","last_fullsized":"2024-03-19T01:23:28.499599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:28.568774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499535+0000","last_change":"2024-03-19T01:23:26.541115+0000","last_active":"2024-03-19T01:23:28.499535+0000","last_peered":"2024-03-19T01:23:28.499535+0000","last_clean":"2024-03-19T01:23:28.499535+0000","last_became_active":"2024-03-19T01:23:26.540408+0000","last_became_peered":"2024-03-19T01:23:26.540408+0000","last_unstale":"2024-03-19T01:23:28.499535+0000","last_undegraded":"2024-03-19T01:23:28.499535+0000","last_fullsized":"2024-03-19T01:23:28.499535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332067+0000","last_change":"2024-03-19T01:23:27.557631+0000","last_active":"2024-03-19T01:23:29.332067+0000","last_peered":"2024-03-19T01:23:29.332067+0000","last_clean":"2024-03-19T01:23:29.332067+0000","last_became_active":"2024-03-19T01:23:27.557324+0000","last_became_peered":"2024-03-19T01:23:27.557324+0000","last_unstale":"2024-03-19T01:23:29.332067+0000","last_undegraded":"2024-03-19T01:23:29.332067+0000","last_fullsized":"2024-03-19T01:23:29.332067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498202+0000","last_change":"2024-03-19T01:23:26.517007+0000","last_active":"2024-03-19T01:23:28.498202+0000","last_peered":"2024-03-19T01:23:28.498202+0000","last_clean":"2024-03-19T01:23:28.498202+0000","last_became_active":"2024-03-19T01:23:26.516710+0000","last_became_peered":"2024-03-19T01:23:26.516710+0000","last_unstale":"2024-03-19T01:23:28.498202+0000","last_undegraded":"2024-03-19T01:23:28.498202+0000","last_fullsized":"2024-03-19T01:23:28.498202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:36.870147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831243+0000","last_change":"2024-03-19T01:23:27.515805+0000","last_active":"2024-03-19T01:23:28.831243+0000","last_peered":"2024-03-19T01:23:28.831243+0000","last_clean":"2024-03-19T01:23:28.831243+0000","last_became_active":"2024-03-19T01:23:27.515099+0000","last_became_peered":"2024-03-19T01:23:27.515099+0000","last_unstale":"2024-03-19T01:23:28.831243+0000","last_undegraded":"2024-03-19T01:23:28.831243+0000","last_fullsized":"2024-03-19T01:23:28.831243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:00.911501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333185+0000","last_change":"2024-03-19T01:23:26.535383+0000","last_active":"2024-03-19T01:23:29.333185+0000","last_peered":"2024-03-19T01:23:29.333185+0000","last_clean":"2024-03-19T01:23:29.333185+0000","last_became_active":"2024-03-19T01:23:26.534052+0000","last_became_peered":"2024-03-19T01:23:26.534052+0000","last_unstale":"2024-03-19T01:23:29.333185+0000","last_undegraded":"2024-03-19T01:23:29.333185+0000","last_fullsized":"2024-03-19T01:23:29.333185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333773+0000","last_change":"2024-03-19T01:23:27.550833+0000","last_active":"2024-03-19T01:23:29.333773+0000","last_peered":"2024-03-19T01:23:29.333773+0000","last_clean":"2024-03-19T01:23:29.333773+0000","last_became_active":"2024-03-19T01:23:27.544992+0000","last_became_peered":"2024-03-19T01:23:27.544992+0000","last_unstale":"2024-03-19T01:23:29.333773+0000","last_undegraded":"2024-03-19T01:23:29.333773+0000","last_fullsized":"2024-03-19T01:23:29.333773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:33.934449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332706+0000","last_change":"2024-03-19T01:23:26.506460+0000","last_active":"2024-03-19T01:23:29.332706+0000","last_peered":"2024-03-19T01:23:29.332706+0000","last_clean":"2024-03-19T01:23:29.332706+0000","last_became_active":"2024-03-19T01:23:26.506155+0000","last_became_peered":"2024-03-19T01:23:26.506155+0000","last_unstale":"2024-03-19T01:23:29.332706+0000","last_undegraded":"2024-03-19T01:23:29.332706+0000","last_fullsized":"2024-03-19T01:23:29.332706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:10.187739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331866+0000","last_change":"2024-03-19T01:23:27.514233+0000","last_active":"2024-03-19T01:23:29.331866+0000","last_peered":"2024-03-19T01:23:29.331866+0000","last_clean":"2024-03-19T01:23:29.331866+0000","last_became_active":"2024-03-19T01:23:27.513950+0000","last_became_peered":"2024-03-19T01:23:27.513950+0000","last_unstale":"2024-03-19T01:23:29.331866+0000","last_undegraded":"2024-03-19T01:23:29.331866+0000","last_fullsized":"2024-03-19T01:23:29.331866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:12:50.425783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499325+0000","last_change":"2024-03-19T01:23:26.516202+0000","last_active":"2024-03-19T01:23:28.499325+0000","last_peered":"2024-03-19T01:23:28.499325+0000","last_clean":"2024-03-19T01:23:28.499325+0000","last_became_active":"2024-03-19T01:23:26.515081+0000","last_became_peered":"2024-03-19T01:23:26.515081+0000","last_unstale":"2024-03-19T01:23:28.499325+0000","last_undegraded":"2024-03-19T01:23:28.499325+0000","last_fullsized":"2024-03-19T01:23:28.499325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332113+0000","last_change":"2024-03-19T01:23:27.534410+0000","last_active":"2024-03-19T01:23:29.332113+0000","last_peered":"2024-03-19T01:23:29.332113+0000","last_clean":"2024-03-19T01:23:29.332113+0000","last_became_active":"2024-03-19T01:23:27.533583+0000","last_became_peered":"2024-03-19T01:23:27.533583+0000","last_unstale":"2024-03-19T01:23:29.332113+0000","last_undegraded":"2024-03-19T01:23:29.332113+0000","last_fullsized":"2024-03-19T01:23:29.332113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:28.568774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332442+0000","last_change":"2024-03-19T01:23:26.530022+0000","last_active":"2024-03-19T01:23:29.332442+0000","last_peered":"2024-03-19T01:23:29.332442+0000","last_clean":"2024-03-19T01:23:29.332442+0000","last_became_active":"2024-03-19T01:23:26.525529+0000","last_became_peered":"2024-03-19T01:23:26.525529+0000","last_unstale":"2024-03-19T01:23:29.332442+0000","last_undegraded":"2024-03-19T01:23:29.332442+0000","last_fullsized":"2024-03-19T01:23:29.332442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332293+0000","last_change":"2024-03-19T01:23:27.542939+0000","last_active":"2024-03-19T01:23:29.332293+0000","last_peered":"2024-03-19T01:23:29.332293+0000","last_clean":"2024-03-19T01:23:29.332293+0000","last_became_active":"2024-03-19T01:23:27.532364+0000","last_became_peered":"2024-03-19T01:23:27.532364+0000","last_unstale":"2024-03-19T01:23:29.332293+0000","last_undegraded":"2024-03-19T01:23:29.332293+0000","last_fullsized":"2024-03-19T01:23:29.332293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572800+0000","last_change":"2024-03-19T01:23:26.513589+0000","last_active":"2024-03-19T01:23:28.572800+0000","last_peered":"2024-03-19T01:23:28.572800+0000","last_clean":"2024-03-19T01:23:28.572800+0000","last_became_active":"2024-03-19T01:23:26.513223+0000","last_became_peered":"2024-03-19T01:23:26.513223+0000","last_unstale":"2024-03-19T01:23:28.572800+0000","last_undegraded":"2024-03-19T01:23:28.572800+0000","last_fullsized":"2024-03-19T01:23:28.572800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331668+0000","last_change":"2024-03-19T01:23:27.552435+0000","last_active":"2024-03-19T01:23:29.331668+0000","last_peered":"2024-03-19T01:23:29.331668+0000","last_clean":"2024-03-19T01:23:29.331668+0000","last_became_active":"2024-03-19T01:23:27.545233+0000","last_became_peered":"2024-03-19T01:23:27.545233+0000","last_unstale":"2024-03-19T01:23:29.331668+0000","last_undegraded":"2024-03-19T01:23:29.331668+0000","last_fullsized":"2024-03-19T01:23:29.331668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831253+0000","last_change":"2024-03-19T01:23:26.515010+0000","last_active":"2024-03-19T01:23:28.831253+0000","last_peered":"2024-03-19T01:23:28.831253+0000","last_clean":"2024-03-19T01:23:28.831253+0000","last_became_active":"2024-03-19T01:23:26.513690+0000","last_became_peered":"2024-03-19T01:23:26.513690+0000","last_unstale":"2024-03-19T01:23:28.831253+0000","last_undegraded":"2024-03-19T01:23:28.831253+0000","last_fullsized":"2024-03-19T01:23:28.831253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:41:56.780781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333245+0000","last_change":"2024-03-19T01:23:27.535739+0000","last_active":"2024-03-19T01:23:29.333245+0000","last_peered":"2024-03-19T01:23:29.333245+0000","last_clean":"2024-03-19T01:23:29.333245+0000","last_became_active":"2024-03-19T01:23:27.534174+0000","last_became_peered":"2024-03-19T01:23:27.534174+0000","last_unstale":"2024-03-19T01:23:29.333245+0000","last_undegraded":"2024-03-19T01:23:29.333245+0000","last_fullsized":"2024-03-19T01:23:29.333245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:01.486458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572994+0000","last_change":"2024-03-19T01:23:26.512861+0000","last_active":"2024-03-19T01:23:28.572994+0000","last_peered":"2024-03-19T01:23:28.572994+0000","last_clean":"2024-03-19T01:23:28.572994+0000","last_became_active":"2024-03-19T01:23:26.512475+0000","last_became_peered":"2024-03-19T01:23:26.512475+0000","last_unstale":"2024-03-19T01:23:28.572994+0000","last_undegraded":"2024-03-19T01:23:28.572994+0000","last_fullsized":"2024-03-19T01:23:28.572994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.334214+0000","last_change":"2024-03-19T01:23:27.553761+0000","last_active":"2024-03-19T01:23:29.334214+0000","last_peered":"2024-03-19T01:23:29.334214+0000","last_clean":"2024-03-19T01:23:29.334214+0000","last_became_active":"2024-03-19T01:23:27.551510+0000","last_became_peered":"2024-03-19T01:23:27.551510+0000","last_unstale":"2024-03-19T01:23:29.334214+0000","last_undegraded":"2024-03-19T01:23:29.334214+0000","last_fullsized":"2024-03-19T01:23:29.334214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:16.398839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498228+0000","last_change":"2024-03-19T01:23:26.520079+0000","last_active":"2024-03-19T01:23:28.498228+0000","last_peered":"2024-03-19T01:23:28.498228+0000","last_clean":"2024-03-19T01:23:28.498228+0000","last_became_active":"2024-03-19T01:23:26.519678+0000","last_became_peered":"2024-03-19T01:23:26.519678+0000","last_unstale":"2024-03-19T01:23:28.498228+0000","last_undegraded":"2024-03-19T01:23:28.498228+0000","last_fullsized":"2024-03-19T01:23:28.498228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:42.729108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499902+0000","last_change":"2024-03-19T01:23:27.535733+0000","last_active":"2024-03-19T01:23:28.499902+0000","last_peered":"2024-03-19T01:23:28.499902+0000","last_clean":"2024-03-19T01:23:28.499902+0000","last_became_active":"2024-03-19T01:23:27.529249+0000","last_became_peered":"2024-03-19T01:23:27.529249+0000","last_unstale":"2024-03-19T01:23:28.499902+0000","last_undegraded":"2024-03-19T01:23:28.499902+0000","last_fullsized":"2024-03-19T01:23:28.499902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:33.934449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572947+0000","last_change":"2024-03-19T01:23:26.518355+0000","last_active":"2024-03-19T01:23:28.572947+0000","last_peered":"2024-03-19T01:23:28.572947+0000","last_clean":"2024-03-19T01:23:28.572947+0000","last_became_active":"2024-03-19T01:23:26.517357+0000","last_became_peered":"2024-03-19T01:23:26.517357+0000","last_unstale":"2024-03-19T01:23:28.572947+0000","last_undegraded":"2024-03-19T01:23:28.572947+0000","last_fullsized":"2024-03-19T01:23:28.572947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333040+0000","last_change":"2024-03-19T01:23:27.564792+0000","last_active":"2024-03-19T01:23:29.333040+0000","last_peered":"2024-03-19T01:23:29.333040+0000","last_clean":"2024-03-19T01:23:29.333040+0000","last_became_active":"2024-03-19T01:23:27.564199+0000","last_became_peered":"2024-03-19T01:23:27.564199+0000","last_unstale":"2024-03-19T01:23:29.333040+0000","last_undegraded":"2024-03-19T01:23:29.333040+0000","last_fullsized":"2024-03-19T01:23:29.333040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:56.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331425+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:23:29.331425+0000","last_peered":"2024-03-19T01:23:29.331425+0000","last_clean":"2024-03-19T01:23:29.331425+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:23:29.331425+0000","last_undegraded":"2024-03-19T01:23:29.331425+0000","last_fullsized":"2024-03-19T01:23:29.331425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498418+0000","last_change":"2024-03-19T01:23:26.516922+0000","last_active":"2024-03-19T01:23:28.498418+0000","last_peered":"2024-03-19T01:23:28.498418+0000","last_clean":"2024-03-19T01:23:28.498418+0000","last_became_active":"2024-03-19T01:23:26.516206+0000","last_became_peered":"2024-03-19T01:23:26.516206+0000","last_unstale":"2024-03-19T01:23:28.498418+0000","last_undegraded":"2024-03-19T01:23:28.498418+0000","last_fullsized":"2024-03-19T01:23:28.498418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569215+0000","last_change":"2024-03-19T01:23:27.557742+0000","last_active":"2024-03-19T01:23:28.569215+0000","last_peered":"2024-03-19T01:23:28.569215+0000","last_clean":"2024-03-19T01:23:28.569215+0000","last_became_active":"2024-03-19T01:23:27.557487+0000","last_became_peered":"2024-03-19T01:23:27.557487+0000","last_unstale":"2024-03-19T01:23:28.569215+0000","last_undegraded":"2024-03-19T01:23:28.569215+0000","last_fullsized":"2024-03-19T01:23:28.569215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331129+0000","last_change":"2024-03-19T01:23:26.539730+0000","last_active":"2024-03-19T01:23:29.331129+0000","last_peered":"2024-03-19T01:23:29.331129+0000","last_clean":"2024-03-19T01:23:29.331129+0000","last_became_active":"2024-03-19T01:23:26.538551+0000","last_became_peered":"2024-03-19T01:23:26.538551+0000","last_unstale":"2024-03-19T01:23:29.331129+0000","last_undegraded":"2024-03-19T01:23:29.331129+0000","last_fullsized":"2024-03-19T01:23:29.331129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331579+0000","last_change":"2024-03-19T01:23:27.524267+0000","last_active":"2024-03-19T01:23:29.331579+0000","last_peered":"2024-03-19T01:23:29.331579+0000","last_clean":"2024-03-19T01:23:29.331579+0000","last_became_active":"2024-03-19T01:23:27.523862+0000","last_became_peered":"2024-03-19T01:23:27.523862+0000","last_unstale":"2024-03-19T01:23:29.331579+0000","last_undegraded":"2024-03-19T01:23:29.331579+0000","last_fullsized":"2024-03-19T01:23:29.331579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:10.543122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572805+0000","last_change":"2024-03-19T01:23:26.516027+0000","last_active":"2024-03-19T01:23:28.572805+0000","last_peered":"2024-03-19T01:23:28.572805+0000","last_clean":"2024-03-19T01:23:28.572805+0000","last_became_active":"2024-03-19T01:23:26.515582+0000","last_became_peered":"2024-03-19T01:23:26.515582+0000","last_unstale":"2024-03-19T01:23:28.572805+0000","last_undegraded":"2024-03-19T01:23:28.572805+0000","last_fullsized":"2024-03-19T01:23:28.572805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498092+0000","last_change":"2024-03-19T01:23:27.515653+0000","last_active":"2024-03-19T01:23:28.498092+0000","last_peered":"2024-03-19T01:23:28.498092+0000","last_clean":"2024-03-19T01:23:28.498092+0000","last_became_active":"2024-03-19T01:23:27.511685+0000","last_became_peered":"2024-03-19T01:23:27.511685+0000","last_unstale":"2024-03-19T01:23:28.498092+0000","last_undegraded":"2024-03-19T01:23:28.498092+0000","last_fullsized":"2024-03-19T01:23:28.498092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499532+0000","last_change":"2024-03-19T01:23:26.511797+0000","last_active":"2024-03-19T01:23:28.499532+0000","last_peered":"2024-03-19T01:23:28.499532+0000","last_clean":"2024-03-19T01:23:28.499532+0000","last_became_active":"2024-03-19T01:23:26.511112+0000","last_became_peered":"2024-03-19T01:23:26.511112+0000","last_unstale":"2024-03-19T01:23:28.499532+0000","last_undegraded":"2024-03-19T01:23:28.499532+0000","last_fullsized":"2024-03-19T01:23:28.499532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830892+0000","last_change":"2024-03-19T01:23:27.523591+0000","last_active":"2024-03-19T01:23:28.830892+0000","last_peered":"2024-03-19T01:23:28.830892+0000","last_clean":"2024-03-19T01:23:28.830892+0000","last_became_active":"2024-03-19T01:23:27.522418+0000","last_became_peered":"2024-03-19T01:23:27.522418+0000","last_unstale":"2024-03-19T01:23:28.830892+0000","last_undegraded":"2024-03-19T01:23:28.830892+0000","last_fullsized":"2024-03-19T01:23:28.830892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498246+0000","last_change":"2024-03-19T01:23:26.519145+0000","last_active":"2024-03-19T01:23:28.498246+0000","last_peered":"2024-03-19T01:23:28.498246+0000","last_clean":"2024-03-19T01:23:28.498246+0000","last_became_active":"2024-03-19T01:23:26.518686+0000","last_became_peered":"2024-03-19T01:23:26.518686+0000","last_unstale":"2024-03-19T01:23:28.498246+0000","last_undegraded":"2024-03-19T01:23:28.498246+0000","last_fullsized":"2024-03-19T01:23:28.498246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499719+0000","last_change":"2024-03-19T01:23:27.549327+0000","last_active":"2024-03-19T01:23:28.499719+0000","last_peered":"2024-03-19T01:23:28.499719+0000","last_clean":"2024-03-19T01:23:28.499719+0000","last_became_active":"2024-03-19T01:23:27.548727+0000","last_became_peered":"2024-03-19T01:23:27.548727+0000","last_unstale":"2024-03-19T01:23:28.499719+0000","last_undegraded":"2024-03-19T01:23:28.499719+0000","last_fullsized":"2024-03-19T01:23:28.499719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:56.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.334028+0000","last_change":"2024-03-19T01:23:26.539153+0000","last_active":"2024-03-19T01:23:29.334028+0000","last_peered":"2024-03-19T01:23:29.334028+0000","last_clean":"2024-03-19T01:23:29.334028+0000","last_became_active":"2024-03-19T01:23:26.537687+0000","last_became_peered":"2024-03-19T01:23:26.537687+0000","last_unstale":"2024-03-19T01:23:29.334028+0000","last_undegraded":"2024-03-19T01:23:29.334028+0000","last_fullsized":"2024-03-19T01:23:29.334028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:59.899930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497941+0000","last_change":"2024-03-19T01:23:27.508732+0000","last_active":"2024-03-19T01:23:28.497941+0000","last_peered":"2024-03-19T01:23:28.497941+0000","last_clean":"2024-03-19T01:23:28.497941+0000","last_became_active":"2024-03-19T01:23:27.508329+0000","last_became_peered":"2024-03-19T01:23:27.508329+0000","last_unstale":"2024-03-19T01:23:28.497941+0000","last_undegraded":"2024-03-19T01:23:28.497941+0000","last_fullsized":"2024-03-19T01:23:28.497941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.920474+0000","last_change":"2024-03-19T01:23:26.536445+0000","last_active":"2024-03-19T01:23:28.920474+0000","last_peered":"2024-03-19T01:23:28.920474+0000","last_clean":"2024-03-19T01:23:28.920474+0000","last_became_active":"2024-03-19T01:23:26.535968+0000","last_became_peered":"2024-03-19T01:23:26.535968+0000","last_unstale":"2024-03-19T01:23:28.920474+0000","last_undegraded":"2024-03-19T01:23:28.920474+0000","last_fullsized":"2024-03-19T01:23:28.920474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569863+0000","last_change":"2024-03-19T01:23:27.553070+0000","last_active":"2024-03-19T01:23:28.569863+0000","last_peered":"2024-03-19T01:23:28.569863+0000","last_clean":"2024-03-19T01:23:28.569863+0000","last_became_active":"2024-03-19T01:23:27.552794+0000","last_became_peered":"2024-03-19T01:23:27.552794+0000","last_unstale":"2024-03-19T01:23:28.569863+0000","last_undegraded":"2024-03-19T01:23:28.569863+0000","last_fullsized":"2024-03-19T01:23:28.569863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333047+0000","last_change":"2024-03-19T01:23:26.537527+0000","last_active":"2024-03-19T01:23:29.333047+0000","last_peered":"2024-03-19T01:23:29.333047+0000","last_clean":"2024-03-19T01:23:29.333047+0000","last_became_active":"2024-03-19T01:23:26.536523+0000","last_became_peered":"2024-03-19T01:23:26.536523+0000","last_unstale":"2024-03-19T01:23:29.333047+0000","last_undegraded":"2024-03-19T01:23:29.333047+0000","last_fullsized":"2024-03-19T01:23:29.333047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332026+0000","last_change":"2024-03-19T01:23:27.511086+0000","last_active":"2024-03-19T01:23:29.332026+0000","last_peered":"2024-03-19T01:23:29.332026+0000","last_clean":"2024-03-19T01:23:29.332026+0000","last_became_active":"2024-03-19T01:23:27.510714+0000","last_became_peered":"2024-03-19T01:23:27.510714+0000","last_unstale":"2024-03-19T01:23:29.332026+0000","last_undegraded":"2024-03-19T01:23:29.332026+0000","last_fullsized":"2024-03-19T01:23:29.332026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331129+0000","last_change":"2024-03-19T01:23:26.539154+0000","last_active":"2024-03-19T01:23:29.331129+0000","last_peered":"2024-03-19T01:23:29.331129+0000","last_clean":"2024-03-19T01:23:29.331129+0000","last_became_active":"2024-03-19T01:23:26.537837+0000","last_became_peered":"2024-03-19T01:23:26.537837+0000","last_unstale":"2024-03-19T01:23:29.331129+0000","last_undegraded":"2024-03-19T01:23:29.331129+0000","last_fullsized":"2024-03-19T01:23:29.331129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830757+0000","last_change":"2024-03-19T01:23:27.512029+0000","last_active":"2024-03-19T01:23:28.830757+0000","last_peered":"2024-03-19T01:23:28.830757+0000","last_clean":"2024-03-19T01:23:28.830757+0000","last_became_active":"2024-03-19T01:23:27.511155+0000","last_became_peered":"2024-03-19T01:23:27.511155+0000","last_unstale":"2024-03-19T01:23:28.830757+0000","last_undegraded":"2024-03-19T01:23:28.830757+0000","last_fullsized":"2024-03-19T01:23:28.830757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:38.911150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497954+0000","last_change":"2024-03-19T01:23:26.528143+0000","last_active":"2024-03-19T01:23:28.497954+0000","last_peered":"2024-03-19T01:23:28.497954+0000","last_clean":"2024-03-19T01:23:28.497954+0000","last_became_active":"2024-03-19T01:23:26.527487+0000","last_became_peered":"2024-03-19T01:23:26.527487+0000","last_unstale":"2024-03-19T01:23:28.497954+0000","last_undegraded":"2024-03-19T01:23:28.497954+0000","last_fullsized":"2024-03-19T01:23:28.497954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:07.823972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569510+0000","last_change":"2024-03-19T01:23:27.518443+0000","last_active":"2024-03-19T01:23:28.569510+0000","last_peered":"2024-03-19T01:23:28.569510+0000","last_clean":"2024-03-19T01:23:28.569510+0000","last_became_active":"2024-03-19T01:23:27.518170+0000","last_became_peered":"2024-03-19T01:23:27.518170+0000","last_unstale":"2024-03-19T01:23:28.569510+0000","last_undegraded":"2024-03-19T01:23:28.569510+0000","last_fullsized":"2024-03-19T01:23:28.569510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498375+0000","last_change":"2024-03-19T01:23:26.511877+0000","last_active":"2024-03-19T01:23:28.498375+0000","last_peered":"2024-03-19T01:23:28.498375+0000","last_clean":"2024-03-19T01:23:28.498375+0000","last_became_active":"2024-03-19T01:23:26.511439+0000","last_became_peered":"2024-03-19T01:23:26.511439+0000","last_unstale":"2024-03-19T01:23:28.498375+0000","last_undegraded":"2024-03-19T01:23:28.498375+0000","last_fullsized":"2024-03-19T01:23:28.498375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569314+0000","last_change":"2024-03-19T01:23:27.523071+0000","last_active":"2024-03-19T01:23:28.569314+0000","last_peered":"2024-03-19T01:23:28.569314+0000","last_clean":"2024-03-19T01:23:28.569314+0000","last_became_active":"2024-03-19T01:23:27.522721+0000","last_became_peered":"2024-03-19T01:23:27.522721+0000","last_unstale":"2024-03-19T01:23:28.569314+0000","last_undegraded":"2024-03-19T01:23:28.569314+0000","last_fullsized":"2024-03-19T01:23:28.569314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332321+0000","last_change":"2024-03-19T01:23:26.543568+0000","last_active":"2024-03-19T01:23:29.332321+0000","last_peered":"2024-03-19T01:23:29.332321+0000","last_clean":"2024-03-19T01:23:29.332321+0000","last_became_active":"2024-03-19T01:23:26.542859+0000","last_became_peered":"2024-03-19T01:23:26.542859+0000","last_unstale":"2024-03-19T01:23:29.332321+0000","last_undegraded":"2024-03-19T01:23:29.332321+0000","last_fullsized":"2024-03-19T01:23:29.332321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:42.729108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332397+0000","last_change":"2024-03-19T01:23:27.547084+0000","last_active":"2024-03-19T01:23:29.332397+0000","last_peered":"2024-03-19T01:23:29.332397+0000","last_clean":"2024-03-19T01:23:29.332397+0000","last_became_active":"2024-03-19T01:23:27.532180+0000","last_became_peered":"2024-03-19T01:23:27.532180+0000","last_unstale":"2024-03-19T01:23:29.332397+0000","last_undegraded":"2024-03-19T01:23:29.332397+0000","last_fullsized":"2024-03-19T01:23:29.332397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499777+0000","last_change":"2024-03-19T01:23:27.557382+0000","last_active":"2024-03-19T01:23:28.499777+0000","last_peered":"2024-03-19T01:23:28.499777+0000","last_clean":"2024-03-19T01:23:28.499777+0000","last_became_active":"2024-03-19T01:23:27.556485+0000","last_became_peered":"2024-03-19T01:23:27.556485+0000","last_unstale":"2024-03-19T01:23:28.499777+0000","last_undegraded":"2024-03-19T01:23:28.499777+0000","last_fullsized":"2024-03-19T01:23:28.499777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:01.486458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332019+0000","last_change":"2024-03-19T01:23:26.541210+0000","last_active":"2024-03-19T01:23:29.332019+0000","last_peered":"2024-03-19T01:23:29.332019+0000","last_clean":"2024-03-19T01:23:29.332019+0000","last_became_active":"2024-03-19T01:23:26.540535+0000","last_became_peered":"2024-03-19T01:23:26.540535+0000","last_unstale":"2024-03-19T01:23:29.332019+0000","last_undegraded":"2024-03-19T01:23:29.332019+0000","last_fullsized":"2024-03-19T01:23:29.332019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:07.823972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499107+0000","last_change":"2024-03-19T01:23:27.547119+0000","last_active":"2024-03-19T01:23:28.499107+0000","last_peered":"2024-03-19T01:23:28.499107+0000","last_clean":"2024-03-19T01:23:28.499107+0000","last_became_active":"2024-03-19T01:23:27.537831+0000","last_became_peered":"2024-03-19T01:23:27.537831+0000","last_unstale":"2024-03-19T01:23:28.499107+0000","last_undegraded":"2024-03-19T01:23:28.499107+0000","last_fullsized":"2024-03-19T01:23:28.499107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333319+0000","last_change":"2024-03-19T01:23:26.544693+0000","last_active":"2024-03-19T01:23:29.333319+0000","last_peered":"2024-03-19T01:23:29.333319+0000","last_clean":"2024-03-19T01:23:29.333319+0000","last_became_active":"2024-03-19T01:23:26.544047+0000","last_became_peered":"2024-03-19T01:23:26.544047+0000","last_unstale":"2024-03-19T01:23:29.333319+0000","last_undegraded":"2024-03-19T01:23:29.333319+0000","last_fullsized":"2024-03-19T01:23:29.333319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:36.870147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331915+0000","last_change":"2024-03-19T01:23:27.549737+0000","last_active":"2024-03-19T01:23:29.331915+0000","last_peered":"2024-03-19T01:23:29.331915+0000","last_clean":"2024-03-19T01:23:29.331915+0000","last_became_active":"2024-03-19T01:23:27.549460+0000","last_became_peered":"2024-03-19T01:23:27.549460+0000","last_unstale":"2024-03-19T01:23:29.331915+0000","last_undegraded":"2024-03-19T01:23:29.331915+0000","last_fullsized":"2024-03-19T01:23:29.331915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569507+0000","last_change":"2024-03-19T01:23:26.520272+0000","last_active":"2024-03-19T01:23:28.569507+0000","last_peered":"2024-03-19T01:23:28.569507+0000","last_clean":"2024-03-19T01:23:28.569507+0000","last_became_active":"2024-03-19T01:23:26.519940+0000","last_became_peered":"2024-03-19T01:23:26.519940+0000","last_unstale":"2024-03-19T01:23:28.569507+0000","last_undegraded":"2024-03-19T01:23:28.569507+0000","last_fullsized":"2024-03-19T01:23:28.569507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:10.187739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569312+0000","last_change":"2024-03-19T01:23:27.541353+0000","last_active":"2024-03-19T01:23:28.569312+0000","last_peered":"2024-03-19T01:23:28.569312+0000","last_clean":"2024-03-19T01:23:28.569312+0000","last_became_active":"2024-03-19T01:23:27.541054+0000","last_became_peered":"2024-03-19T01:23:27.541054+0000","last_unstale":"2024-03-19T01:23:28.569312+0000","last_undegraded":"2024-03-19T01:23:28.569312+0000","last_fullsized":"2024-03-19T01:23:28.569312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332192+0000","last_change":"2024-03-19T01:23:26.528906+0000","last_active":"2024-03-19T01:23:29.332192+0000","last_peered":"2024-03-19T01:23:29.332192+0000","last_clean":"2024-03-19T01:23:29.332192+0000","last_became_active":"2024-03-19T01:23:26.528592+0000","last_became_peered":"2024-03-19T01:23:26.528592+0000","last_unstale":"2024-03-19T01:23:29.332192+0000","last_undegraded":"2024-03-19T01:23:29.332192+0000","last_fullsized":"2024-03-19T01:23:29.332192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:23:36.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:23:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.399+0000 7f2e21c95640 1 Processor -- start 2024-03-19T01:23:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.399+0000 7f2e21c95640 1 -- start start 2024-03-19T01:23:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e21c95640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c06df50 0x7f2e1c06e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e21c95640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2e1c147d10 0x7f2e1c148170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e21c95640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e1c0aec10 con 0x7f2e1c0af510 2024-03-19T01:23:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e21c95640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f2e1c06be90 con 0x7f2e1c147d10 2024-03-19T01:23:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e21c95640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1c09f440 con 0x7f2e1c06df50 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2e1c0af510 0x7f2e1c0af9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43256/0 (socket says 172.21.15.7:43256) 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1bfff640 1 -- 172.21.15.7:0/607578207 learned_addr learned my addr 172.21.15.7:0/607578207 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1b7fe640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c06df50 0x7f2e1c06e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1affd640 1 --2- 172.21.15.7:0/607578207 >> v2:172.21.15.167:3300/0 conn(0x7f2e1c147d10 0x7f2e1c148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3149437260 0 0) 0x7f2e1c0aec10 con 0x7f2e1c0af510 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e04003530 con 0x7f2e1c0af510 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4205688941 0 0) 0x7f2e04003530 con 0x7f2e1c0af510 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c06df50 msgr2=0x7f2e1c06e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c06df50 0x7f2e1c06e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 >> v2:172.21.15.167:3300/0 conn(0x7f2e1c147d10 msgr2=0x7f2e1c148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 --2- 172.21.15.7:0/607578207 >> v2:172.21.15.167:3300/0 conn(0x7f2e1c147d10 0x7f2e1c148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c06c0e0 con 0x7f2e1c0af510 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f2e14001a00 con 0x7f2e1c0af510 2024-03-19T01:23:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 >> v1:172.21.15.7:6789/0 conn(0x7f2e1c0af510 legacy=0x7f2e1c0af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e040039c0 0x7f2e04003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e04004810 0x7f2e04004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c06df50 0x7f2e040051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c06be90 con 0x7f2e040039c0 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c09f440 con 0x7f2e04004810 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2e0400ba50 con 0x7f2e1c06df50 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1b7fe640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e040039c0 0x7f2e04003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1bfff640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c06df50 0x7f2e040051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1b7fe640 1 -- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c06df50 msgr2=0x7f2e040051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1b7fe640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c06df50 0x7f2e040051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1affd640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e04004810 0x7f2e04004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1b7fe640 1 -- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e04004810 msgr2=0x7f2e04004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1b7fe640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e04004810 0x7f2e04004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1b7fe640 1 -- 172.21.15.7:0/607578207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e04003530 con 0x7f2e040039c0 2024-03-19T01:23:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1b7fe640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e040039c0 0x7f2e04003de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009ae0 tx=0x7f2e1001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7eca40042577a17 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e100b2070 con 0x7f2e040039c0 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e100a5510 con 0x7f2e040039c0 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.403+0000 7f2e1a7fc640 1 -- 172.21.15.7:0/607578207 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e100096b0 con 0x7f2e040039c0 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 -- 172.21.15.7:0/607578207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e040039c0 msgr2=0x7f2e04003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 --2- 172.21.15.7:0/607578207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e040039c0 0x7f2e04003de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009ae0 tx=0x7f2e1001ca70 comp rx=0 tx=0).stop 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1bfff640 1 -- 172.21.15.7:0/607578207 reap_dead start 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 -- 172.21.15.7:0/607578207 shutdown_connections 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 -- 172.21.15.7:0/607578207 >> 172.21.15.7:0/607578207 conn(0x7f2e1c0b6510 msgr2=0x7f2e1c0b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 -- 172.21.15.7:0/607578207 shutdown_connections 2024-03-19T01:23:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 -- 172.21.15.7:0/607578207 wait complete. 2024-03-19T01:23:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 Processor -- start 2024-03-19T01:23:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 -- start start 2024-03-19T01:23:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e1c071600 0x7f2e1c0b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e1c0b3cc0 0x7f2e1c141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c142160 0x7f2e1c14edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c06d160 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c06cfe0 con 0x7f2e1c071600 2024-03-19T01:23:36.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e21c95640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1c1525a0 con 0x7f2e1c142160 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e1c0b3cc0 0x7f2e1c141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e1c0b3cc0 0x7f2e1c141c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40222/0 (socket says 172.21.15.7:40222) 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1affd640 1 -- 172.21.15.7:0/602242652 learned_addr learned my addr 172.21.15.7:0/602242652 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1b7fe640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e1c071600 0x7f2e1c0b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1bfff640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c142160 0x7f2e1c14edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1affd640 1 -- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c142160 msgr2=0x7f2e1c14edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1affd640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c142160 0x7f2e1c14edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1affd640 1 -- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e1c071600 msgr2=0x7f2e1c0b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1affd640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e1c071600 0x7f2e1c0b1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1affd640 1 -- 172.21.15.7:0/602242652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c0b3530 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1b7fe640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e1c071600 0x7f2e1c0b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1bfff640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c142160 0x7f2e1c14edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.407+0000 7f2e1affd640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e1c0b3cc0 0x7f2e1c141c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c010900 tx=0x7f2e0c010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e03fff640 1 -- 172.21.15.7:0/602242652 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c00f040 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e03fff640 1 -- 172.21.15.7:0/602242652 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e0c00ec20 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1c142610 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e03fff640 1 -- 172.21.15.7:0/602242652 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c01c400 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e1c0b1050 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e1c06c350 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e03fff640 1 -- 172.21.15.7:0/602242652 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2e0c01c5a0 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e03fff640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2e08067660 0x7f2e08069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e03fff640 1 -- 172.21.15.7:0/602242652 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2e0c085e00 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e1b7fe640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2e08067660 0x7f2e08069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.411+0000 7f2e1b7fe640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2e08067660 0x7f2e08069b20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e100af000 tx=0x7f2e10008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:36.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.415+0000 7f2e03fff640 1 -- 172.21.15.7:0/602242652 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2e0c018030 con 0x7f2e1c0b3cc0 2024-03-19T01:23:36.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.567+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2e1c075460 con 0x7f2e08067660 2024-03-19T01:23:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.571+0000 7f2e03fff640 1 -- 172.21.15.7:0/602242652 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f2e1c075460 con 0x7f2e08067660 2024-03-19T01:23:36.580 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:36.582 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:23:36.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2e08067660 msgr2=0x7f2e08069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2e08067660 0x7f2e08069b20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e100af000 tx=0x7f2e10008040 comp rx=0 tx=0).stop 2024-03-19T01:23:36.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e1c0b3cc0 msgr2=0x7f2e1c141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e1c0b3cc0 0x7f2e1c141c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c010900 tx=0x7f2e0c010dc0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 shutdown_connections 2024-03-19T01:23:36.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2e08067660 0x7f2e08069b20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e1c142160 0x7f2e1c14edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e1c0b3cc0 0x7f2e1c141c20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 --2- 172.21.15.7:0/602242652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e1c071600 0x7f2e1c0b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 >> 172.21.15.7:0/602242652 conn(0x7f2e1c0b6510 msgr2=0x7f2e1c144ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:36.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 shutdown_connections 2024-03-19T01:23:36.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.575+0000 7f2e21c95640 1 -- 172.21.15.7:0/602242652 wait complete. 2024-03-19T01:23:36.597 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-19T01:23:34.892348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235756,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738964,"statfs":{"total":773094113280,"available":772852699136,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002750"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333975+0000","last_change":"2024-03-19T01:23:27.551581+0000","last_active":"2024-03-19T01:23:29.333975+0000","last_peered":"2024-03-19T01:23:29.333975+0000","last_clean":"2024-03-19T01:23:29.333975+0000","last_became_active":"2024-03-19T01:23:27.550909+0000","last_became_peered":"2024-03-19T01:23:27.550909+0000","last_unstale":"2024-03-19T01:23:29.333975+0000","last_undegraded":"2024-03-19T01:23:29.333975+0000","last_fullsized":"2024-03-19T01:23:29.333975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:14.092827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498885+0000","last_change":"2024-03-19T01:23:27.512582+0000","last_active":"2024-03-19T01:23:28.498885+0000","last_peered":"2024-03-19T01:23:28.498885+0000","last_clean":"2024-03-19T01:23:28.498885+0000","last_became_active":"2024-03-19T01:23:27.512252+0000","last_became_peered":"2024-03-19T01:23:27.512252+0000","last_unstale":"2024-03-19T01:23:28.498885+0000","last_undegraded":"2024-03-19T01:23:28.498885+0000","last_fullsized":"2024-03-19T01:23:28.498885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331920+0000","last_change":"2024-03-19T01:23:27.512676+0000","last_active":"2024-03-19T01:23:29.331920+0000","last_peered":"2024-03-19T01:23:29.331920+0000","last_clean":"2024-03-19T01:23:29.331920+0000","last_became_active":"2024-03-19T01:23:27.512311+0000","last_became_peered":"2024-03-19T01:23:27.512311+0000","last_unstale":"2024-03-19T01:23:29.331920+0000","last_undegraded":"2024-03-19T01:23:29.331920+0000","last_fullsized":"2024-03-19T01:23:29.331920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498730+0000","last_change":"2024-03-19T01:23:27.538857+0000","last_active":"2024-03-19T01:23:28.498730+0000","last_peered":"2024-03-19T01:23:28.498730+0000","last_clean":"2024-03-19T01:23:28.498730+0000","last_became_active":"2024-03-19T01:23:27.536808+0000","last_became_peered":"2024-03-19T01:23:27.536808+0000","last_unstale":"2024-03-19T01:23:28.498730+0000","last_undegraded":"2024-03-19T01:23:28.498730+0000","last_fullsized":"2024-03-19T01:23:28.498730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332035+0000","last_change":"2024-03-19T01:23:27.543517+0000","last_active":"2024-03-19T01:23:29.332035+0000","last_peered":"2024-03-19T01:23:29.332035+0000","last_clean":"2024-03-19T01:23:29.332035+0000","last_became_active":"2024-03-19T01:23:27.542861+0000","last_became_peered":"2024-03-19T01:23:27.542861+0000","last_unstale":"2024-03-19T01:23:29.332035+0000","last_undegraded":"2024-03-19T01:23:29.332035+0000","last_fullsized":"2024-03-19T01:23:29.332035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498639+0000","last_change":"2024-03-19T01:23:27.549333+0000","last_active":"2024-03-19T01:23:28.498639+0000","last_peered":"2024-03-19T01:23:28.498639+0000","last_clean":"2024-03-19T01:23:28.498639+0000","last_became_active":"2024-03-19T01:23:27.548338+0000","last_became_peered":"2024-03-19T01:23:27.548338+0000","last_unstale":"2024-03-19T01:23:28.498639+0000","last_undegraded":"2024-03-19T01:23:28.498639+0000","last_fullsized":"2024-03-19T01:23:28.498639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331558+0000","last_change":"2024-03-19T01:23:27.536206+0000","last_active":"2024-03-19T01:23:29.331558+0000","last_peered":"2024-03-19T01:23:29.331558+0000","last_clean":"2024-03-19T01:23:29.331558+0000","last_became_active":"2024-03-19T01:23:27.535402+0000","last_became_peered":"2024-03-19T01:23:27.535402+0000","last_unstale":"2024-03-19T01:23:29.331558+0000","last_undegraded":"2024-03-19T01:23:29.331558+0000","last_fullsized":"2024-03-19T01:23:29.331558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569176+0000","last_change":"2024-03-19T01:23:27.548444+0000","last_active":"2024-03-19T01:23:28.569176+0000","last_peered":"2024-03-19T01:23:28.569176+0000","last_clean":"2024-03-19T01:23:28.569176+0000","last_became_active":"2024-03-19T01:23:27.548160+0000","last_became_peered":"2024-03-19T01:23:27.548160+0000","last_unstale":"2024-03-19T01:23:28.569176+0000","last_undegraded":"2024-03-19T01:23:28.569176+0000","last_fullsized":"2024-03-19T01:23:28.569176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.334001+0000","last_change":"2024-03-19T01:23:27.558139+0000","last_active":"2024-03-19T01:23:29.334001+0000","last_peered":"2024-03-19T01:23:29.334001+0000","last_clean":"2024-03-19T01:23:29.334001+0000","last_became_active":"2024-03-19T01:23:27.557564+0000","last_became_peered":"2024-03-19T01:23:27.557564+0000","last_unstale":"2024-03-19T01:23:29.334001+0000","last_undegraded":"2024-03-19T01:23:29.334001+0000","last_fullsized":"2024-03-19T01:23:29.334001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:31:46.209018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333331+0000","last_change":"2024-03-19T01:23:27.549847+0000","last_active":"2024-03-19T01:23:29.333331+0000","last_peered":"2024-03-19T01:23:29.333331+0000","last_clean":"2024-03-19T01:23:29.333331+0000","last_became_active":"2024-03-19T01:23:27.548601+0000","last_became_peered":"2024-03-19T01:23:27.548601+0000","last_unstale":"2024-03-19T01:23:29.333331+0000","last_undegraded":"2024-03-19T01:23:29.333331+0000","last_fullsized":"2024-03-19T01:23:29.333331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333016+0000","last_change":"2024-03-19T01:23:27.558581+0000","last_active":"2024-03-19T01:23:29.333016+0000","last_peered":"2024-03-19T01:23:29.333016+0000","last_clean":"2024-03-19T01:23:29.333016+0000","last_became_active":"2024-03-19T01:23:27.558044+0000","last_became_peered":"2024-03-19T01:23:27.558044+0000","last_unstale":"2024-03-19T01:23:29.333016+0000","last_undegraded":"2024-03-19T01:23:29.333016+0000","last_fullsized":"2024-03-19T01:23:29.333016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831030+0000","last_change":"2024-03-19T01:23:27.543968+0000","last_active":"2024-03-19T01:23:28.831030+0000","last_peered":"2024-03-19T01:23:28.831030+0000","last_clean":"2024-03-19T01:23:28.831030+0000","last_became_active":"2024-03-19T01:23:27.543453+0000","last_became_peered":"2024-03-19T01:23:27.543453+0000","last_unstale":"2024-03-19T01:23:28.831030+0000","last_undegraded":"2024-03-19T01:23:28.831030+0000","last_fullsized":"2024-03-19T01:23:28.831030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569582+0000","last_change":"2024-03-19T01:23:27.541445+0000","last_active":"2024-03-19T01:23:28.569582+0000","last_peered":"2024-03-19T01:23:28.569582+0000","last_clean":"2024-03-19T01:23:28.569582+0000","last_became_active":"2024-03-19T01:23:27.541162+0000","last_became_peered":"2024-03-19T01:23:27.541162+0000","last_unstale":"2024-03-19T01:23:28.569582+0000","last_undegraded":"2024-03-19T01:23:28.569582+0000","last_fullsized":"2024-03-19T01:23:28.569582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331530+0000","last_change":"2024-03-19T01:23:27.515384+0000","last_active":"2024-03-19T01:23:29.331530+0000","last_peered":"2024-03-19T01:23:29.331530+0000","last_clean":"2024-03-19T01:23:29.331530+0000","last_became_active":"2024-03-19T01:23:27.514935+0000","last_became_peered":"2024-03-19T01:23:27.514935+0000","last_unstale":"2024-03-19T01:23:29.331530+0000","last_undegraded":"2024-03-19T01:23:29.331530+0000","last_fullsized":"2024-03-19T01:23:29.331530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:52.938759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497668+0000","last_change":"2024-03-19T01:23:27.509553+0000","last_active":"2024-03-19T01:23:28.497668+0000","last_peered":"2024-03-19T01:23:28.497668+0000","last_clean":"2024-03-19T01:23:28.497668+0000","last_became_active":"2024-03-19T01:23:27.509178+0000","last_became_peered":"2024-03-19T01:23:27.509178+0000","last_unstale":"2024-03-19T01:23:28.497668+0000","last_undegraded":"2024-03-19T01:23:28.497668+0000","last_fullsized":"2024-03-19T01:23:28.497668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498214+0000","last_change":"2024-03-19T01:23:27.547056+0000","last_active":"2024-03-19T01:23:28.498214+0000","last_peered":"2024-03-19T01:23:28.498214+0000","last_clean":"2024-03-19T01:23:28.498214+0000","last_became_active":"2024-03-19T01:23:27.538339+0000","last_became_peered":"2024-03-19T01:23:27.538339+0000","last_unstale":"2024-03-19T01:23:28.498214+0000","last_undegraded":"2024-03-19T01:23:28.498214+0000","last_fullsized":"2024-03-19T01:23:28.498214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.500126+0000","last_change":"2024-03-19T01:23:27.562551+0000","last_active":"2024-03-19T01:23:28.500126+0000","last_peered":"2024-03-19T01:23:28.500126+0000","last_clean":"2024-03-19T01:23:28.500126+0000","last_became_active":"2024-03-19T01:23:27.561996+0000","last_became_peered":"2024-03-19T01:23:27.561996+0000","last_unstale":"2024-03-19T01:23:28.500126+0000","last_undegraded":"2024-03-19T01:23:28.500126+0000","last_fullsized":"2024-03-19T01:23:28.500126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:27.331497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572895+0000","last_change":"2024-03-19T01:23:27.541038+0000","last_active":"2024-03-19T01:23:28.572895+0000","last_peered":"2024-03-19T01:23:28.572895+0000","last_clean":"2024-03-19T01:23:28.572895+0000","last_became_active":"2024-03-19T01:23:27.540751+0000","last_became_peered":"2024-03-19T01:23:27.540751+0000","last_unstale":"2024-03-19T01:23:28.572895+0000","last_undegraded":"2024-03-19T01:23:28.572895+0000","last_fullsized":"2024-03-19T01:23:28.572895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:37.881718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333787+0000","last_change":"2024-03-19T01:23:27.552053+0000","last_active":"2024-03-19T01:23:29.333787+0000","last_peered":"2024-03-19T01:23:29.333787+0000","last_clean":"2024-03-19T01:23:29.333787+0000","last_became_active":"2024-03-19T01:23:27.551405+0000","last_became_peered":"2024-03-19T01:23:27.551405+0000","last_unstale":"2024-03-19T01:23:29.333787+0000","last_undegraded":"2024-03-19T01:23:29.333787+0000","last_fullsized":"2024-03-19T01:23:29.333787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:27.331497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332923+0000","last_change":"2024-03-19T01:23:27.548149+0000","last_active":"2024-03-19T01:23:29.332923+0000","last_peered":"2024-03-19T01:23:29.332923+0000","last_clean":"2024-03-19T01:23:29.332923+0000","last_became_active":"2024-03-19T01:23:27.542105+0000","last_became_peered":"2024-03-19T01:23:27.542105+0000","last_unstale":"2024-03-19T01:23:29.332923+0000","last_undegraded":"2024-03-19T01:23:29.332923+0000","last_fullsized":"2024-03-19T01:23:29.332923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332234+0000","last_change":"2024-03-19T01:23:27.521814+0000","last_active":"2024-03-19T01:23:29.332234+0000","last_peered":"2024-03-19T01:23:29.332234+0000","last_clean":"2024-03-19T01:23:29.332234+0000","last_became_active":"2024-03-19T01:23:27.519639+0000","last_became_peered":"2024-03-19T01:23:27.519639+0000","last_unstale":"2024-03-19T01:23:29.332234+0000","last_undegraded":"2024-03-19T01:23:29.332234+0000","last_fullsized":"2024-03-19T01:23:29.332234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499387+0000","last_change":"2024-03-19T01:23:27.558248+0000","last_active":"2024-03-19T01:23:28.499387+0000","last_peered":"2024-03-19T01:23:28.499387+0000","last_clean":"2024-03-19T01:23:28.499387+0000","last_became_active":"2024-03-19T01:23:27.557663+0000","last_became_peered":"2024-03-19T01:23:27.557663+0000","last_unstale":"2024-03-19T01:23:28.499387+0000","last_undegraded":"2024-03-19T01:23:28.499387+0000","last_fullsized":"2024-03-19T01:23:28.499387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.568928+0000","last_change":"2024-03-19T01:23:27.532119+0000","last_active":"2024-03-19T01:23:28.568928+0000","last_peered":"2024-03-19T01:23:28.568928+0000","last_clean":"2024-03-19T01:23:28.568928+0000","last_became_active":"2024-03-19T01:23:27.531846+0000","last_became_peered":"2024-03-19T01:23:27.531846+0000","last_unstale":"2024-03-19T01:23:28.568928+0000","last_undegraded":"2024-03-19T01:23:28.568928+0000","last_fullsized":"2024-03-19T01:23:28.568928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498373+0000","last_change":"2024-03-19T01:23:27.546292+0000","last_active":"2024-03-19T01:23:28.498373+0000","last_peered":"2024-03-19T01:23:28.498373+0000","last_clean":"2024-03-19T01:23:28.498373+0000","last_became_active":"2024-03-19T01:23:27.537369+0000","last_became_peered":"2024-03-19T01:23:27.537369+0000","last_unstale":"2024-03-19T01:23:28.498373+0000","last_undegraded":"2024-03-19T01:23:28.498373+0000","last_fullsized":"2024-03-19T01:23:28.498373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569601+0000","last_change":"2024-03-19T01:23:27.522448+0000","last_active":"2024-03-19T01:23:28.569601+0000","last_peered":"2024-03-19T01:23:28.569601+0000","last_clean":"2024-03-19T01:23:28.569601+0000","last_became_active":"2024-03-19T01:23:27.522201+0000","last_became_peered":"2024-03-19T01:23:27.522201+0000","last_unstale":"2024-03-19T01:23:28.569601+0000","last_undegraded":"2024-03-19T01:23:28.569601+0000","last_fullsized":"2024-03-19T01:23:28.569601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333402+0000","last_change":"2024-03-19T01:23:27.551178+0000","last_active":"2024-03-19T01:23:29.333402+0000","last_peered":"2024-03-19T01:23:29.333402+0000","last_clean":"2024-03-19T01:23:29.333402+0000","last_became_active":"2024-03-19T01:23:27.550571+0000","last_became_peered":"2024-03-19T01:23:27.550571+0000","last_unstale":"2024-03-19T01:23:29.333402+0000","last_undegraded":"2024-03-19T01:23:29.333402+0000","last_fullsized":"2024-03-19T01:23:29.333402+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497894+0000","last_change":"2024-03-19T01:23:27.509194+0000","last_active":"2024-03-19T01:23:28.497894+0000","last_peered":"2024-03-19T01:23:28.497894+0000","last_clean":"2024-03-19T01:23:28.497894+0000","last_became_active":"2024-03-19T01:23:27.508796+0000","last_became_peered":"2024-03-19T01:23:27.508796+0000","last_unstale":"2024-03-19T01:23:28.497894+0000","last_undegraded":"2024-03-19T01:23:28.497894+0000","last_fullsized":"2024-03-19T01:23:28.497894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569258+0000","last_change":"2024-03-19T01:23:27.522380+0000","last_active":"2024-03-19T01:23:28.569258+0000","last_peered":"2024-03-19T01:23:28.569258+0000","last_clean":"2024-03-19T01:23:28.569258+0000","last_became_active":"2024-03-19T01:23:27.521968+0000","last_became_peered":"2024-03-19T01:23:27.521968+0000","last_unstale":"2024-03-19T01:23:28.569258+0000","last_undegraded":"2024-03-19T01:23:28.569258+0000","last_fullsized":"2024-03-19T01:23:28.569258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:55.388319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332566+0000","last_change":"2024-03-19T01:23:27.516056+0000","last_active":"2024-03-19T01:23:29.332566+0000","last_peered":"2024-03-19T01:23:29.332566+0000","last_clean":"2024-03-19T01:23:29.332566+0000","last_became_active":"2024-03-19T01:23:27.515651+0000","last_became_peered":"2024-03-19T01:23:27.515651+0000","last_unstale":"2024-03-19T01:23:29.332566+0000","last_undegraded":"2024-03-19T01:23:29.332566+0000","last_fullsized":"2024-03-19T01:23:29.332566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499052+0000","last_change":"2024-03-19T01:23:27.547751+0000","last_active":"2024-03-19T01:23:28.499052+0000","last_peered":"2024-03-19T01:23:28.499052+0000","last_clean":"2024-03-19T01:23:28.499052+0000","last_became_active":"2024-03-19T01:23:27.530027+0000","last_became_peered":"2024-03-19T01:23:27.530027+0000","last_unstale":"2024-03-19T01:23:28.499052+0000","last_undegraded":"2024-03-19T01:23:28.499052+0000","last_fullsized":"2024-03-19T01:23:28.499052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:55.801423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572767+0000","last_change":"2024-03-19T01:23:27.538411+0000","last_active":"2024-03-19T01:23:28.572767+0000","last_peered":"2024-03-19T01:23:28.572767+0000","last_clean":"2024-03-19T01:23:28.572767+0000","last_became_active":"2024-03-19T01:23:27.537893+0000","last_became_peered":"2024-03-19T01:23:27.537893+0000","last_unstale":"2024-03-19T01:23:28.572767+0000","last_undegraded":"2024-03-19T01:23:28.572767+0000","last_fullsized":"2024-03-19T01:23:28.572767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331763+0000","last_change":"2024-03-19T01:23:27.539735+0000","last_active":"2024-03-19T01:23:29.331763+0000","last_peered":"2024-03-19T01:23:29.331763+0000","last_clean":"2024-03-19T01:23:29.331763+0000","last_became_active":"2024-03-19T01:23:27.539022+0000","last_became_peered":"2024-03-19T01:23:27.539022+0000","last_unstale":"2024-03-19T01:23:29.331763+0000","last_undegraded":"2024-03-19T01:23:29.331763+0000","last_fullsized":"2024-03-19T01:23:29.331763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:35.922159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830822+0000","last_change":"2024-03-19T01:23:27.503767+0000","last_active":"2024-03-19T01:23:28.830822+0000","last_peered":"2024-03-19T01:23:28.830822+0000","last_clean":"2024-03-19T01:23:28.830822+0000","last_became_active":"2024-03-19T01:23:27.503030+0000","last_became_peered":"2024-03-19T01:23:27.503030+0000","last_unstale":"2024-03-19T01:23:28.830822+0000","last_undegraded":"2024-03-19T01:23:28.830822+0000","last_fullsized":"2024-03-19T01:23:28.830822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:08.835543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569515+0000","last_change":"2024-03-19T01:23:26.532832+0000","last_active":"2024-03-19T01:23:28.569515+0000","last_peered":"2024-03-19T01:23:28.569515+0000","last_clean":"2024-03-19T01:23:28.569515+0000","last_became_active":"2024-03-19T01:23:26.532516+0000","last_became_peered":"2024-03-19T01:23:26.532516+0000","last_unstale":"2024-03-19T01:23:28.569515+0000","last_undegraded":"2024-03-19T01:23:28.569515+0000","last_fullsized":"2024-03-19T01:23:28.569515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332652+0000","last_change":"2024-03-19T01:23:27.522605+0000","last_active":"2024-03-19T01:23:29.332652+0000","last_peered":"2024-03-19T01:23:29.332652+0000","last_clean":"2024-03-19T01:23:29.332652+0000","last_became_active":"2024-03-19T01:23:27.522344+0000","last_became_peered":"2024-03-19T01:23:27.522344+0000","last_unstale":"2024-03-19T01:23:29.332652+0000","last_undegraded":"2024-03-19T01:23:29.332652+0000","last_fullsized":"2024-03-19T01:23:29.332652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332682+0000","last_change":"2024-03-19T01:23:26.535891+0000","last_active":"2024-03-19T01:23:29.332682+0000","last_peered":"2024-03-19T01:23:29.332682+0000","last_clean":"2024-03-19T01:23:29.332682+0000","last_became_active":"2024-03-19T01:23:26.535218+0000","last_became_peered":"2024-03-19T01:23:26.535218+0000","last_unstale":"2024-03-19T01:23:29.332682+0000","last_undegraded":"2024-03-19T01:23:29.332682+0000","last_fullsized":"2024-03-19T01:23:29.332682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.568845+0000","last_change":"2024-03-19T01:23:26.536647+0000","last_active":"2024-03-19T01:23:28.568845+0000","last_peered":"2024-03-19T01:23:28.568845+0000","last_clean":"2024-03-19T01:23:28.568845+0000","last_became_active":"2024-03-19T01:23:26.535919+0000","last_became_peered":"2024-03-19T01:23:26.535919+0000","last_unstale":"2024-03-19T01:23:28.568845+0000","last_undegraded":"2024-03-19T01:23:28.568845+0000","last_fullsized":"2024-03-19T01:23:28.568845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569044+0000","last_change":"2024-03-19T01:23:27.517889+0000","last_active":"2024-03-19T01:23:28.569044+0000","last_peered":"2024-03-19T01:23:28.569044+0000","last_clean":"2024-03-19T01:23:28.569044+0000","last_became_active":"2024-03-19T01:23:27.515221+0000","last_became_peered":"2024-03-19T01:23:27.515221+0000","last_unstale":"2024-03-19T01:23:28.569044+0000","last_undegraded":"2024-03-19T01:23:28.569044+0000","last_fullsized":"2024-03-19T01:23:28.569044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499348+0000","last_change":"2024-03-19T01:23:26.518817+0000","last_active":"2024-03-19T01:23:28.499348+0000","last_peered":"2024-03-19T01:23:28.499348+0000","last_clean":"2024-03-19T01:23:28.499348+0000","last_became_active":"2024-03-19T01:23:26.518008+0000","last_became_peered":"2024-03-19T01:23:26.518008+0000","last_unstale":"2024-03-19T01:23:28.499348+0000","last_undegraded":"2024-03-19T01:23:28.499348+0000","last_fullsized":"2024-03-19T01:23:28.499348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498152+0000","last_change":"2024-03-19T01:23:27.514550+0000","last_active":"2024-03-19T01:23:28.498152+0000","last_peered":"2024-03-19T01:23:28.498152+0000","last_clean":"2024-03-19T01:23:28.498152+0000","last_became_active":"2024-03-19T01:23:27.512624+0000","last_became_peered":"2024-03-19T01:23:27.512624+0000","last_unstale":"2024-03-19T01:23:28.498152+0000","last_undegraded":"2024-03-19T01:23:28.498152+0000","last_fullsized":"2024-03-19T01:23:28.498152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332747+0000","last_change":"2024-03-19T01:23:26.536461+0000","last_active":"2024-03-19T01:23:29.332747+0000","last_peered":"2024-03-19T01:23:29.332747+0000","last_clean":"2024-03-19T01:23:29.332747+0000","last_became_active":"2024-03-19T01:23:26.535707+0000","last_became_peered":"2024-03-19T01:23:26.535707+0000","last_unstale":"2024-03-19T01:23:29.332747+0000","last_undegraded":"2024-03-19T01:23:29.332747+0000","last_fullsized":"2024-03-19T01:23:29.332747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499599+0000","last_change":"2024-03-19T01:23:27.561680+0000","last_active":"2024-03-19T01:23:28.499599+0000","last_peered":"2024-03-19T01:23:28.499599+0000","last_clean":"2024-03-19T01:23:28.499599+0000","last_became_active":"2024-03-19T01:23:27.559667+0000","last_became_peered":"2024-03-19T01:23:27.559667+0000","last_unstale":"2024-03-19T01:23:28.499599+0000","last_undegraded":"2024-03-19T01:23:28.499599+0000","last_fullsized":"2024-03-19T01:23:28.499599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:28.568774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499535+0000","last_change":"2024-03-19T01:23:26.541115+0000","last_active":"2024-03-19T01:23:28.499535+0000","last_peered":"2024-03-19T01:23:28.499535+0000","last_clean":"2024-03-19T01:23:28.499535+0000","last_became_active":"2024-03-19T01:23:26.540408+0000","last_became_peered":"2024-03-19T01:23:26.540408+0000","last_unstale":"2024-03-19T01:23:28.499535+0000","last_undegraded":"2024-03-19T01:23:28.499535+0000","last_fullsized":"2024-03-19T01:23:28.499535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332067+0000","last_change":"2024-03-19T01:23:27.557631+0000","last_active":"2024-03-19T01:23:29.332067+0000","last_peered":"2024-03-19T01:23:29.332067+0000","last_clean":"2024-03-19T01:23:29.332067+0000","last_became_active":"2024-03-19T01:23:27.557324+0000","last_became_peered":"2024-03-19T01:23:27.557324+0000","last_unstale":"2024-03-19T01:23:29.332067+0000","last_undegraded":"2024-03-19T01:23:29.332067+0000","last_fullsized":"2024-03-19T01:23:29.332067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498202+0000","last_change":"2024-03-19T01:23:26.517007+0000","last_active":"2024-03-19T01:23:28.498202+0000","last_peered":"2024-03-19T01:23:28.498202+0000","last_clean":"2024-03-19T01:23:28.498202+0000","last_became_active":"2024-03-19T01:23:26.516710+0000","last_became_peered":"2024-03-19T01:23:26.516710+0000","last_unstale":"2024-03-19T01:23:28.498202+0000","last_undegraded":"2024-03-19T01:23:28.498202+0000","last_fullsized":"2024-03-19T01:23:28.498202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:36.870147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831243+0000","last_change":"2024-03-19T01:23:27.515805+0000","last_active":"2024-03-19T01:23:28.831243+0000","last_peered":"2024-03-19T01:23:28.831243+0000","last_clean":"2024-03-19T01:23:28.831243+0000","last_became_active":"2024-03-19T01:23:27.515099+0000","last_became_peered":"2024-03-19T01:23:27.515099+0000","last_unstale":"2024-03-19T01:23:28.831243+0000","last_undegraded":"2024-03-19T01:23:28.831243+0000","last_fullsized":"2024-03-19T01:23:28.831243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:00.911501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333185+0000","last_change":"2024-03-19T01:23:26.535383+0000","last_active":"2024-03-19T01:23:29.333185+0000","last_peered":"2024-03-19T01:23:29.333185+0000","last_clean":"2024-03-19T01:23:29.333185+0000","last_became_active":"2024-03-19T01:23:26.534052+0000","last_became_peered":"2024-03-19T01:23:26.534052+0000","last_unstale":"2024-03-19T01:23:29.333185+0000","last_undegraded":"2024-03-19T01:23:29.333185+0000","last_fullsized":"2024-03-19T01:23:29.333185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333773+0000","last_change":"2024-03-19T01:23:27.550833+0000","last_active":"2024-03-19T01:23:29.333773+0000","last_peered":"2024-03-19T01:23:29.333773+0000","last_clean":"2024-03-19T01:23:29.333773+0000","last_became_active":"2024-03-19T01:23:27.544992+0000","last_became_peered":"2024-03-19T01:23:27.544992+0000","last_unstale":"2024-03-19T01:23:29.333773+0000","last_undegraded":"2024-03-19T01:23:29.333773+0000","last_fullsized":"2024-03-19T01:23:29.333773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:33.934449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332706+0000","last_change":"2024-03-19T01:23:26.506460+0000","last_active":"2024-03-19T01:23:29.332706+0000","last_peered":"2024-03-19T01:23:29.332706+0000","last_clean":"2024-03-19T01:23:29.332706+0000","last_became_active":"2024-03-19T01:23:26.506155+0000","last_became_peered":"2024-03-19T01:23:26.506155+0000","last_unstale":"2024-03-19T01:23:29.332706+0000","last_undegraded":"2024-03-19T01:23:29.332706+0000","last_fullsized":"2024-03-19T01:23:29.332706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:10.187739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331866+0000","last_change":"2024-03-19T01:23:27.514233+0000","last_active":"2024-03-19T01:23:29.331866+0000","last_peered":"2024-03-19T01:23:29.331866+0000","last_clean":"2024-03-19T01:23:29.331866+0000","last_became_active":"2024-03-19T01:23:27.513950+0000","last_became_peered":"2024-03-19T01:23:27.513950+0000","last_unstale":"2024-03-19T01:23:29.331866+0000","last_undegraded":"2024-03-19T01:23:29.331866+0000","last_fullsized":"2024-03-19T01:23:29.331866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:12:50.425783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499325+0000","last_change":"2024-03-19T01:23:26.516202+0000","last_active":"2024-03-19T01:23:28.499325+0000","last_peered":"2024-03-19T01:23:28.499325+0000","last_clean":"2024-03-19T01:23:28.499325+0000","last_became_active":"2024-03-19T01:23:26.515081+0000","last_became_peered":"2024-03-19T01:23:26.515081+0000","last_unstale":"2024-03-19T01:23:28.499325+0000","last_undegraded":"2024-03-19T01:23:28.499325+0000","last_fullsized":"2024-03-19T01:23:28.499325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332113+0000","last_change":"2024-03-19T01:23:27.534410+0000","last_active":"2024-03-19T01:23:29.332113+0000","last_peered":"2024-03-19T01:23:29.332113+0000","last_clean":"2024-03-19T01:23:29.332113+0000","last_became_active":"2024-03-19T01:23:27.533583+0000","last_became_peered":"2024-03-19T01:23:27.533583+0000","last_unstale":"2024-03-19T01:23:29.332113+0000","last_undegraded":"2024-03-19T01:23:29.332113+0000","last_fullsized":"2024-03-19T01:23:29.332113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:28.568774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332442+0000","last_change":"2024-03-19T01:23:26.530022+0000","last_active":"2024-03-19T01:23:29.332442+0000","last_peered":"2024-03-19T01:23:29.332442+0000","last_clean":"2024-03-19T01:23:29.332442+0000","last_became_active":"2024-03-19T01:23:26.525529+0000","last_became_peered":"2024-03-19T01:23:26.525529+0000","last_unstale":"2024-03-19T01:23:29.332442+0000","last_undegraded":"2024-03-19T01:23:29.332442+0000","last_fullsized":"2024-03-19T01:23:29.332442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332293+0000","last_change":"2024-03-19T01:23:27.542939+0000","last_active":"2024-03-19T01:23:29.332293+0000","last_peered":"2024-03-19T01:23:29.332293+0000","last_clean":"2024-03-19T01:23:29.332293+0000","last_became_active":"2024-03-19T01:23:27.532364+0000","last_became_peered":"2024-03-19T01:23:27.532364+0000","last_unstale":"2024-03-19T01:23:29.332293+0000","last_undegraded":"2024-03-19T01:23:29.332293+0000","last_fullsized":"2024-03-19T01:23:29.332293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572800+0000","last_change":"2024-03-19T01:23:26.513589+0000","last_active":"2024-03-19T01:23:28.572800+0000","last_peered":"2024-03-19T01:23:28.572800+0000","last_clean":"2024-03-19T01:23:28.572800+0000","last_became_active":"2024-03-19T01:23:26.513223+0000","last_became_peered":"2024-03-19T01:23:26.513223+0000","last_unstale":"2024-03-19T01:23:28.572800+0000","last_undegraded":"2024-03-19T01:23:28.572800+0000","last_fullsized":"2024-03-19T01:23:28.572800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331668+0000","last_change":"2024-03-19T01:23:27.552435+0000","last_active":"2024-03-19T01:23:29.331668+0000","last_peered":"2024-03-19T01:23:29.331668+0000","last_clean":"2024-03-19T01:23:29.331668+0000","last_became_active":"2024-03-19T01:23:27.545233+0000","last_became_peered":"2024-03-19T01:23:27.545233+0000","last_unstale":"2024-03-19T01:23:29.331668+0000","last_undegraded":"2024-03-19T01:23:29.331668+0000","last_fullsized":"2024-03-19T01:23:29.331668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.831253+0000","last_change":"2024-03-19T01:23:26.515010+0000","last_active":"2024-03-19T01:23:28.831253+0000","last_peered":"2024-03-19T01:23:28.831253+0000","last_clean":"2024-03-19T01:23:28.831253+0000","last_became_active":"2024-03-19T01:23:26.513690+0000","last_became_peered":"2024-03-19T01:23:26.513690+0000","last_unstale":"2024-03-19T01:23:28.831253+0000","last_undegraded":"2024-03-19T01:23:28.831253+0000","last_fullsized":"2024-03-19T01:23:28.831253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:41:56.780781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333245+0000","last_change":"2024-03-19T01:23:27.535739+0000","last_active":"2024-03-19T01:23:29.333245+0000","last_peered":"2024-03-19T01:23:29.333245+0000","last_clean":"2024-03-19T01:23:29.333245+0000","last_became_active":"2024-03-19T01:23:27.534174+0000","last_became_peered":"2024-03-19T01:23:27.534174+0000","last_unstale":"2024-03-19T01:23:29.333245+0000","last_undegraded":"2024-03-19T01:23:29.333245+0000","last_fullsized":"2024-03-19T01:23:29.333245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:01.486458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572994+0000","last_change":"2024-03-19T01:23:26.512861+0000","last_active":"2024-03-19T01:23:28.572994+0000","last_peered":"2024-03-19T01:23:28.572994+0000","last_clean":"2024-03-19T01:23:28.572994+0000","last_became_active":"2024-03-19T01:23:26.512475+0000","last_became_peered":"2024-03-19T01:23:26.512475+0000","last_unstale":"2024-03-19T01:23:28.572994+0000","last_undegraded":"2024-03-19T01:23:28.572994+0000","last_fullsized":"2024-03-19T01:23:28.572994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.334214+0000","last_change":"2024-03-19T01:23:27.553761+0000","last_active":"2024-03-19T01:23:29.334214+0000","last_peered":"2024-03-19T01:23:29.334214+0000","last_clean":"2024-03-19T01:23:29.334214+0000","last_became_active":"2024-03-19T01:23:27.551510+0000","last_became_peered":"2024-03-19T01:23:27.551510+0000","last_unstale":"2024-03-19T01:23:29.334214+0000","last_undegraded":"2024-03-19T01:23:29.334214+0000","last_fullsized":"2024-03-19T01:23:29.334214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:16.398839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498228+0000","last_change":"2024-03-19T01:23:26.520079+0000","last_active":"2024-03-19T01:23:28.498228+0000","last_peered":"2024-03-19T01:23:28.498228+0000","last_clean":"2024-03-19T01:23:28.498228+0000","last_became_active":"2024-03-19T01:23:26.519678+0000","last_became_peered":"2024-03-19T01:23:26.519678+0000","last_unstale":"2024-03-19T01:23:28.498228+0000","last_undegraded":"2024-03-19T01:23:28.498228+0000","last_fullsized":"2024-03-19T01:23:28.498228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:42.729108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499902+0000","last_change":"2024-03-19T01:23:27.535733+0000","last_active":"2024-03-19T01:23:28.499902+0000","last_peered":"2024-03-19T01:23:28.499902+0000","last_clean":"2024-03-19T01:23:28.499902+0000","last_became_active":"2024-03-19T01:23:27.529249+0000","last_became_peered":"2024-03-19T01:23:27.529249+0000","last_unstale":"2024-03-19T01:23:28.499902+0000","last_undegraded":"2024-03-19T01:23:28.499902+0000","last_fullsized":"2024-03-19T01:23:28.499902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:33.934449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572947+0000","last_change":"2024-03-19T01:23:26.518355+0000","last_active":"2024-03-19T01:23:28.572947+0000","last_peered":"2024-03-19T01:23:28.572947+0000","last_clean":"2024-03-19T01:23:28.572947+0000","last_became_active":"2024-03-19T01:23:26.517357+0000","last_became_peered":"2024-03-19T01:23:26.517357+0000","last_unstale":"2024-03-19T01:23:28.572947+0000","last_undegraded":"2024-03-19T01:23:28.572947+0000","last_fullsized":"2024-03-19T01:23:28.572947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333040+0000","last_change":"2024-03-19T01:23:27.564792+0000","last_active":"2024-03-19T01:23:29.333040+0000","last_peered":"2024-03-19T01:23:29.333040+0000","last_clean":"2024-03-19T01:23:29.333040+0000","last_became_active":"2024-03-19T01:23:27.564199+0000","last_became_peered":"2024-03-19T01:23:27.564199+0000","last_unstale":"2024-03-19T01:23:29.333040+0000","last_undegraded":"2024-03-19T01:23:29.333040+0000","last_fullsized":"2024-03-19T01:23:29.333040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:56.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331425+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:23:29.331425+0000","last_peered":"2024-03-19T01:23:29.331425+0000","last_clean":"2024-03-19T01:23:29.331425+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:23:29.331425+0000","last_undegraded":"2024-03-19T01:23:29.331425+0000","last_fullsized":"2024-03-19T01:23:29.331425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498418+0000","last_change":"2024-03-19T01:23:26.516922+0000","last_active":"2024-03-19T01:23:28.498418+0000","last_peered":"2024-03-19T01:23:28.498418+0000","last_clean":"2024-03-19T01:23:28.498418+0000","last_became_active":"2024-03-19T01:23:26.516206+0000","last_became_peered":"2024-03-19T01:23:26.516206+0000","last_unstale":"2024-03-19T01:23:28.498418+0000","last_undegraded":"2024-03-19T01:23:28.498418+0000","last_fullsized":"2024-03-19T01:23:28.498418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:14.532541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569215+0000","last_change":"2024-03-19T01:23:27.557742+0000","last_active":"2024-03-19T01:23:28.569215+0000","last_peered":"2024-03-19T01:23:28.569215+0000","last_clean":"2024-03-19T01:23:28.569215+0000","last_became_active":"2024-03-19T01:23:27.557487+0000","last_became_peered":"2024-03-19T01:23:27.557487+0000","last_unstale":"2024-03-19T01:23:28.569215+0000","last_undegraded":"2024-03-19T01:23:28.569215+0000","last_fullsized":"2024-03-19T01:23:28.569215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331129+0000","last_change":"2024-03-19T01:23:26.539730+0000","last_active":"2024-03-19T01:23:29.331129+0000","last_peered":"2024-03-19T01:23:29.331129+0000","last_clean":"2024-03-19T01:23:29.331129+0000","last_became_active":"2024-03-19T01:23:26.538551+0000","last_became_peered":"2024-03-19T01:23:26.538551+0000","last_unstale":"2024-03-19T01:23:29.331129+0000","last_undegraded":"2024-03-19T01:23:29.331129+0000","last_fullsized":"2024-03-19T01:23:29.331129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331579+0000","last_change":"2024-03-19T01:23:27.524267+0000","last_active":"2024-03-19T01:23:29.331579+0000","last_peered":"2024-03-19T01:23:29.331579+0000","last_clean":"2024-03-19T01:23:29.331579+0000","last_became_active":"2024-03-19T01:23:27.523862+0000","last_became_peered":"2024-03-19T01:23:27.523862+0000","last_unstale":"2024-03-19T01:23:29.331579+0000","last_undegraded":"2024-03-19T01:23:29.331579+0000","last_fullsized":"2024-03-19T01:23:29.331579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:10.543122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.572805+0000","last_change":"2024-03-19T01:23:26.516027+0000","last_active":"2024-03-19T01:23:28.572805+0000","last_peered":"2024-03-19T01:23:28.572805+0000","last_clean":"2024-03-19T01:23:28.572805+0000","last_became_active":"2024-03-19T01:23:26.515582+0000","last_became_peered":"2024-03-19T01:23:26.515582+0000","last_unstale":"2024-03-19T01:23:28.572805+0000","last_undegraded":"2024-03-19T01:23:28.572805+0000","last_fullsized":"2024-03-19T01:23:28.572805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498092+0000","last_change":"2024-03-19T01:23:27.515653+0000","last_active":"2024-03-19T01:23:28.498092+0000","last_peered":"2024-03-19T01:23:28.498092+0000","last_clean":"2024-03-19T01:23:28.498092+0000","last_became_active":"2024-03-19T01:23:27.511685+0000","last_became_peered":"2024-03-19T01:23:27.511685+0000","last_unstale":"2024-03-19T01:23:28.498092+0000","last_undegraded":"2024-03-19T01:23:28.498092+0000","last_fullsized":"2024-03-19T01:23:28.498092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499532+0000","last_change":"2024-03-19T01:23:26.511797+0000","last_active":"2024-03-19T01:23:28.499532+0000","last_peered":"2024-03-19T01:23:28.499532+0000","last_clean":"2024-03-19T01:23:28.499532+0000","last_became_active":"2024-03-19T01:23:26.511112+0000","last_became_peered":"2024-03-19T01:23:26.511112+0000","last_unstale":"2024-03-19T01:23:28.499532+0000","last_undegraded":"2024-03-19T01:23:28.499532+0000","last_fullsized":"2024-03-19T01:23:28.499532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830892+0000","last_change":"2024-03-19T01:23:27.523591+0000","last_active":"2024-03-19T01:23:28.830892+0000","last_peered":"2024-03-19T01:23:28.830892+0000","last_clean":"2024-03-19T01:23:28.830892+0000","last_became_active":"2024-03-19T01:23:27.522418+0000","last_became_peered":"2024-03-19T01:23:27.522418+0000","last_unstale":"2024-03-19T01:23:28.830892+0000","last_undegraded":"2024-03-19T01:23:28.830892+0000","last_fullsized":"2024-03-19T01:23:28.830892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:43.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498246+0000","last_change":"2024-03-19T01:23:26.519145+0000","last_active":"2024-03-19T01:23:28.498246+0000","last_peered":"2024-03-19T01:23:28.498246+0000","last_clean":"2024-03-19T01:23:28.498246+0000","last_became_active":"2024-03-19T01:23:26.518686+0000","last_became_peered":"2024-03-19T01:23:26.518686+0000","last_unstale":"2024-03-19T01:23:28.498246+0000","last_undegraded":"2024-03-19T01:23:28.498246+0000","last_fullsized":"2024-03-19T01:23:28.498246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499719+0000","last_change":"2024-03-19T01:23:27.549327+0000","last_active":"2024-03-19T01:23:28.499719+0000","last_peered":"2024-03-19T01:23:28.499719+0000","last_clean":"2024-03-19T01:23:28.499719+0000","last_became_active":"2024-03-19T01:23:27.548727+0000","last_became_peered":"2024-03-19T01:23:27.548727+0000","last_unstale":"2024-03-19T01:23:28.499719+0000","last_undegraded":"2024-03-19T01:23:28.499719+0000","last_fullsized":"2024-03-19T01:23:28.499719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:56.015936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.334028+0000","last_change":"2024-03-19T01:23:26.539153+0000","last_active":"2024-03-19T01:23:29.334028+0000","last_peered":"2024-03-19T01:23:29.334028+0000","last_clean":"2024-03-19T01:23:29.334028+0000","last_became_active":"2024-03-19T01:23:26.537687+0000","last_became_peered":"2024-03-19T01:23:26.537687+0000","last_unstale":"2024-03-19T01:23:29.334028+0000","last_undegraded":"2024-03-19T01:23:29.334028+0000","last_fullsized":"2024-03-19T01:23:29.334028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:59.899930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497941+0000","last_change":"2024-03-19T01:23:27.508732+0000","last_active":"2024-03-19T01:23:28.497941+0000","last_peered":"2024-03-19T01:23:28.497941+0000","last_clean":"2024-03-19T01:23:28.497941+0000","last_became_active":"2024-03-19T01:23:27.508329+0000","last_became_peered":"2024-03-19T01:23:27.508329+0000","last_unstale":"2024-03-19T01:23:28.497941+0000","last_undegraded":"2024-03-19T01:23:28.497941+0000","last_fullsized":"2024-03-19T01:23:28.497941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.920474+0000","last_change":"2024-03-19T01:23:26.536445+0000","last_active":"2024-03-19T01:23:28.920474+0000","last_peered":"2024-03-19T01:23:28.920474+0000","last_clean":"2024-03-19T01:23:28.920474+0000","last_became_active":"2024-03-19T01:23:26.535968+0000","last_became_peered":"2024-03-19T01:23:26.535968+0000","last_unstale":"2024-03-19T01:23:28.920474+0000","last_undegraded":"2024-03-19T01:23:28.920474+0000","last_fullsized":"2024-03-19T01:23:28.920474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:37.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569863+0000","last_change":"2024-03-19T01:23:27.553070+0000","last_active":"2024-03-19T01:23:28.569863+0000","last_peered":"2024-03-19T01:23:28.569863+0000","last_clean":"2024-03-19T01:23:28.569863+0000","last_became_active":"2024-03-19T01:23:27.552794+0000","last_became_peered":"2024-03-19T01:23:27.552794+0000","last_unstale":"2024-03-19T01:23:28.569863+0000","last_undegraded":"2024-03-19T01:23:28.569863+0000","last_fullsized":"2024-03-19T01:23:28.569863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:16.607609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333047+0000","last_change":"2024-03-19T01:23:26.537527+0000","last_active":"2024-03-19T01:23:29.333047+0000","last_peered":"2024-03-19T01:23:29.333047+0000","last_clean":"2024-03-19T01:23:29.333047+0000","last_became_active":"2024-03-19T01:23:26.536523+0000","last_became_peered":"2024-03-19T01:23:26.536523+0000","last_unstale":"2024-03-19T01:23:29.333047+0000","last_undegraded":"2024-03-19T01:23:29.333047+0000","last_fullsized":"2024-03-19T01:23:29.333047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:30.271387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332026+0000","last_change":"2024-03-19T01:23:27.511086+0000","last_active":"2024-03-19T01:23:29.332026+0000","last_peered":"2024-03-19T01:23:29.332026+0000","last_clean":"2024-03-19T01:23:29.332026+0000","last_became_active":"2024-03-19T01:23:27.510714+0000","last_became_peered":"2024-03-19T01:23:27.510714+0000","last_unstale":"2024-03-19T01:23:29.332026+0000","last_undegraded":"2024-03-19T01:23:29.332026+0000","last_fullsized":"2024-03-19T01:23:29.332026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:03.320251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331129+0000","last_change":"2024-03-19T01:23:26.539154+0000","last_active":"2024-03-19T01:23:29.331129+0000","last_peered":"2024-03-19T01:23:29.331129+0000","last_clean":"2024-03-19T01:23:29.331129+0000","last_became_active":"2024-03-19T01:23:26.537837+0000","last_became_peered":"2024-03-19T01:23:26.537837+0000","last_unstale":"2024-03-19T01:23:29.331129+0000","last_undegraded":"2024-03-19T01:23:29.331129+0000","last_fullsized":"2024-03-19T01:23:29.331129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:53.899157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.830757+0000","last_change":"2024-03-19T01:23:27.512029+0000","last_active":"2024-03-19T01:23:28.830757+0000","last_peered":"2024-03-19T01:23:28.830757+0000","last_clean":"2024-03-19T01:23:28.830757+0000","last_became_active":"2024-03-19T01:23:27.511155+0000","last_became_peered":"2024-03-19T01:23:27.511155+0000","last_unstale":"2024-03-19T01:23:28.830757+0000","last_undegraded":"2024-03-19T01:23:28.830757+0000","last_fullsized":"2024-03-19T01:23:28.830757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:38.911150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.497954+0000","last_change":"2024-03-19T01:23:26.528143+0000","last_active":"2024-03-19T01:23:28.497954+0000","last_peered":"2024-03-19T01:23:28.497954+0000","last_clean":"2024-03-19T01:23:28.497954+0000","last_became_active":"2024-03-19T01:23:26.527487+0000","last_became_peered":"2024-03-19T01:23:26.527487+0000","last_unstale":"2024-03-19T01:23:28.497954+0000","last_undegraded":"2024-03-19T01:23:28.497954+0000","last_fullsized":"2024-03-19T01:23:28.497954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:07.823972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569510+0000","last_change":"2024-03-19T01:23:27.518443+0000","last_active":"2024-03-19T01:23:28.569510+0000","last_peered":"2024-03-19T01:23:28.569510+0000","last_clean":"2024-03-19T01:23:28.569510+0000","last_became_active":"2024-03-19T01:23:27.518170+0000","last_became_peered":"2024-03-19T01:23:27.518170+0000","last_unstale":"2024-03-19T01:23:28.569510+0000","last_undegraded":"2024-03-19T01:23:28.569510+0000","last_fullsized":"2024-03-19T01:23:28.569510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:47:50.686128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.498375+0000","last_change":"2024-03-19T01:23:26.511877+0000","last_active":"2024-03-19T01:23:28.498375+0000","last_peered":"2024-03-19T01:23:28.498375+0000","last_clean":"2024-03-19T01:23:28.498375+0000","last_became_active":"2024-03-19T01:23:26.511439+0000","last_became_peered":"2024-03-19T01:23:26.511439+0000","last_unstale":"2024-03-19T01:23:28.498375+0000","last_undegraded":"2024-03-19T01:23:28.498375+0000","last_fullsized":"2024-03-19T01:23:28.498375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:47:50.098329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569314+0000","last_change":"2024-03-19T01:23:27.523071+0000","last_active":"2024-03-19T01:23:28.569314+0000","last_peered":"2024-03-19T01:23:28.569314+0000","last_clean":"2024-03-19T01:23:28.569314+0000","last_became_active":"2024-03-19T01:23:27.522721+0000","last_became_peered":"2024-03-19T01:23:27.522721+0000","last_unstale":"2024-03-19T01:23:28.569314+0000","last_undegraded":"2024-03-19T01:23:28.569314+0000","last_fullsized":"2024-03-19T01:23:28.569314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:01.895989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332321+0000","last_change":"2024-03-19T01:23:26.543568+0000","last_active":"2024-03-19T01:23:29.332321+0000","last_peered":"2024-03-19T01:23:29.332321+0000","last_clean":"2024-03-19T01:23:29.332321+0000","last_became_active":"2024-03-19T01:23:26.542859+0000","last_became_peered":"2024-03-19T01:23:26.542859+0000","last_unstale":"2024-03-19T01:23:29.332321+0000","last_undegraded":"2024-03-19T01:23:29.332321+0000","last_fullsized":"2024-03-19T01:23:29.332321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:42.729108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332397+0000","last_change":"2024-03-19T01:23:27.547084+0000","last_active":"2024-03-19T01:23:29.332397+0000","last_peered":"2024-03-19T01:23:29.332397+0000","last_clean":"2024-03-19T01:23:29.332397+0000","last_became_active":"2024-03-19T01:23:27.532180+0000","last_became_peered":"2024-03-19T01:23:27.532180+0000","last_unstale":"2024-03-19T01:23:29.332397+0000","last_undegraded":"2024-03-19T01:23:29.332397+0000","last_fullsized":"2024-03-19T01:23:29.332397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:36.692412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499777+0000","last_change":"2024-03-19T01:23:27.557382+0000","last_active":"2024-03-19T01:23:28.499777+0000","last_peered":"2024-03-19T01:23:28.499777+0000","last_clean":"2024-03-19T01:23:28.499777+0000","last_became_active":"2024-03-19T01:23:27.556485+0000","last_became_peered":"2024-03-19T01:23:27.556485+0000","last_unstale":"2024-03-19T01:23:28.499777+0000","last_undegraded":"2024-03-19T01:23:28.499777+0000","last_fullsized":"2024-03-19T01:23:28.499777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:01.486458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332019+0000","last_change":"2024-03-19T01:23:26.541210+0000","last_active":"2024-03-19T01:23:29.332019+0000","last_peered":"2024-03-19T01:23:29.332019+0000","last_clean":"2024-03-19T01:23:29.332019+0000","last_became_active":"2024-03-19T01:23:26.540535+0000","last_became_peered":"2024-03-19T01:23:26.540535+0000","last_unstale":"2024-03-19T01:23:29.332019+0000","last_undegraded":"2024-03-19T01:23:29.332019+0000","last_fullsized":"2024-03-19T01:23:29.332019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:07.823972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.499107+0000","last_change":"2024-03-19T01:23:27.547119+0000","last_active":"2024-03-19T01:23:28.499107+0000","last_peered":"2024-03-19T01:23:28.499107+0000","last_clean":"2024-03-19T01:23:28.499107+0000","last_became_active":"2024-03-19T01:23:27.537831+0000","last_became_peered":"2024-03-19T01:23:27.537831+0000","last_unstale":"2024-03-19T01:23:28.499107+0000","last_undegraded":"2024-03-19T01:23:28.499107+0000","last_fullsized":"2024-03-19T01:23:28.499107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:09.291239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.333319+0000","last_change":"2024-03-19T01:23:26.544693+0000","last_active":"2024-03-19T01:23:29.333319+0000","last_peered":"2024-03-19T01:23:29.333319+0000","last_clean":"2024-03-19T01:23:29.333319+0000","last_became_active":"2024-03-19T01:23:26.544047+0000","last_became_peered":"2024-03-19T01:23:26.544047+0000","last_unstale":"2024-03-19T01:23:29.333319+0000","last_undegraded":"2024-03-19T01:23:29.333319+0000","last_fullsized":"2024-03-19T01:23:29.333319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:36.870147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.331915+0000","last_change":"2024-03-19T01:23:27.549737+0000","last_active":"2024-03-19T01:23:29.331915+0000","last_peered":"2024-03-19T01:23:29.331915+0000","last_clean":"2024-03-19T01:23:29.331915+0000","last_became_active":"2024-03-19T01:23:27.549460+0000","last_became_peered":"2024-03-19T01:23:27.549460+0000","last_unstale":"2024-03-19T01:23:29.331915+0000","last_undegraded":"2024-03-19T01:23:29.331915+0000","last_fullsized":"2024-03-19T01:23:29.331915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:26.824013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569507+0000","last_change":"2024-03-19T01:23:26.520272+0000","last_active":"2024-03-19T01:23:28.569507+0000","last_peered":"2024-03-19T01:23:28.569507+0000","last_clean":"2024-03-19T01:23:28.569507+0000","last_became_active":"2024-03-19T01:23:26.519940+0000","last_became_peered":"2024-03-19T01:23:26.519940+0000","last_unstale":"2024-03-19T01:23:28.569507+0000","last_undegraded":"2024-03-19T01:23:28.569507+0000","last_fullsized":"2024-03-19T01:23:28.569507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:10.187739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:28.569312+0000","last_change":"2024-03-19T01:23:27.541353+0000","last_active":"2024-03-19T01:23:28.569312+0000","last_peered":"2024-03-19T01:23:28.569312+0000","last_clean":"2024-03-19T01:23:28.569312+0000","last_became_active":"2024-03-19T01:23:27.541054+0000","last_became_peered":"2024-03-19T01:23:27.541054+0000","last_unstale":"2024-03-19T01:23:28.569312+0000","last_undegraded":"2024-03-19T01:23:28.569312+0000","last_fullsized":"2024-03-19T01:23:28.569312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:26.476770+0000","last_clean_scrub_stamp":"2024-03-19T01:23:26.476770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:23.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T01:23:29.332192+0000","last_change":"2024-03-19T01:23:26.528906+0000","last_active":"2024-03-19T01:23:29.332192+0000","last_peered":"2024-03-19T01:23:29.332192+0000","last_clean":"2024-03-19T01:23:29.332192+0000","last_became_active":"2024-03-19T01:23:26.528592+0000","last_became_peered":"2024-03-19T01:23:26.528592+0000","last_unstale":"2024-03-19T01:23:29.332192+0000","last_undegraded":"2024-03-19T01:23:29.332192+0000","last_fullsized":"2024-03-19T01:23:29.332192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:25.465199+0000","last_clean_scrub_stamp":"2024-03-19T01:23:25.465199+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:36:05.623577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:23:36.600 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-19T01:23:36.601 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-19T01:23:36.601 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-19T01:23:36.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-19T01:23:36.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.739+0000 7fbe86250640 1 Processor -- start 2024-03-19T01:23:36.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.739+0000 7fbe86250640 1 -- start start 2024-03-19T01:23:36.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.739+0000 7fbe86250640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fbe8006d5d0 0x7fbe8013e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.739+0000 7fbe86250640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe800725b0 con 0x7fbe8013e690 2024-03-19T01:23:36.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.739+0000 7fbe86250640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe8006d9b0 con 0x7fbe800b5af0 2024-03-19T01:23:36.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.739+0000 7fbe86250640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fbe800b5f60 con 0x7fbe8006d5d0 2024-03-19T01:23:36.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7effd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbe8013e690 0x7fbe80147720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43264/0 (socket says 172.21.15.7:43264) 2024-03-19T01:23:36.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7effd640 1 -- 172.21.15.7:0/425831324 learned_addr learned my addr 172.21.15.7:0/425831324 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:36.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7f7fe640 1 --2- 172.21.15.7:0/425831324 >> v2:172.21.15.167:3300/0 conn(0x7fbe8006d5d0 0x7fbe8013e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1764030052 0 0) 0x7fbe800725b0 con 0x7fbe8013e690 2024-03-19T01:23:36.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe68003530 con 0x7fbe8013e690 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3274392252 0 0) 0x7fbe68003530 con 0x7fbe8013e690 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 >> v2:172.21.15.167:3300/0 conn(0x7fbe8006d5d0 msgr2=0x7fbe8013e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 --2- 172.21.15.7:0/425831324 >> v2:172.21.15.167:3300/0 conn(0x7fbe8006d5d0 0x7fbe8013e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 >> v1:172.21.15.167:6789/0 conn(0x7fbe800b5af0 legacy=0x7fbe8014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe800b61b0 con 0x7fbe8013e690 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fbe74001680 con 0x7fbe8013e690 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 >> v1:172.21.15.7:6789/0 conn(0x7fbe8013e690 legacy=0x7fbe80147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe680039c0 0x7fbe68003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe68004810 0x7fbe68004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe680051d0 0x7fbe6800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbe800b5f60 con 0x7fbe680051d0 2024-03-19T01:23:36.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6800c070 con 0x7fbe68004810 2024-03-19T01:23:36.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbe6800c270 con 0x7fbe680039c0 2024-03-19T01:23:36.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7ffff640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe680051d0 0x7fbe6800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7effd640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe68004810 0x7fbe68004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7f7fe640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe680039c0 0x7fbe68003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7ffff640 1 -- 172.21.15.7:0/425831324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe680039c0 msgr2=0x7fbe68003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7ffff640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe680039c0 0x7fbe68003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7ffff640 1 -- 172.21.15.7:0/425831324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe68004810 msgr2=0x7fbe68004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:36.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7ffff640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe68004810 0x7fbe68004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7ffff640 1 -- 172.21.15.7:0/425831324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe68003530 con 0x7fbe680051d0 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7ffff640 1 -- 172.21.15.7:0/425831324 reap_dead start 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7ffff640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe680051d0 0x7fbe6800b9d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbe78001e70 tx=0x7fbe7801c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23feaea83aff1cc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78001a90 con 0x7fbe680051d0 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe78003bc0 con 0x7fbe680051d0 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.743+0000 7fbe7e7fc640 1 -- 172.21.15.7:0/425831324 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78005020 con 0x7fbe680051d0 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- 172.21.15.7:0/425831324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe680051d0 msgr2=0x7fbe6800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe680051d0 0x7fbe6800b9d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbe78001e70 tx=0x7fbe7801c780 comp rx=0 tx=0).stop 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- 172.21.15.7:0/425831324 shutdown_connections 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 --2- 172.21.15.7:0/425831324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe680051d0 0x7fbe6800b9d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- 172.21.15.7:0/425831324 >> 172.21.15.7:0/425831324 conn(0x7fbe800b68e0 msgr2=0x7fbe800b6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- 172.21.15.7:0/425831324 shutdown_connections 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- 172.21.15.7:0/425831324 wait complete. 2024-03-19T01:23:36.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 Processor -- start 2024-03-19T01:23:36.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- start start 2024-03-19T01:23:36.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe680051d0 0x7fbe800af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe800ae780 0x7fbe801e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe801e09e0 0x7fbe80138d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbe80072780 con 0x7fbe801e09e0 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbe80071470 con 0x7fbe680051d0 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbe80072cb0 con 0x7fbe800ae780 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe801e09e0 0x7fbe80138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe801e09e0 0x7fbe80138d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40240/0 (socket says 172.21.15.7:40240) 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7ffff640 1 -- 172.21.15.7:0/939905829 learned_addr learned my addr 172.21.15.7:0/939905829 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7effd640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe800ae780 0x7fbe801e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7f7fe640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe680051d0 0x7fbe800af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7ffff640 1 -- 172.21.15.7:0/939905829 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe800ae780 msgr2=0x7fbe801e04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7ffff640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe800ae780 0x7fbe801e04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7ffff640 1 -- 172.21.15.7:0/939905829 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe680051d0 msgr2=0x7fbe800af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7ffff640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe680051d0 0x7fbe800af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7ffff640 1 -- 172.21.15.7:0/939905829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8013ebb0 con 0x7fbe801e09e0 2024-03-19T01:23:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe7ffff640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe801e09e0 0x7fbe80138d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbe7801c5b0 tx=0x7fbe68005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe67fff640 1 -- 172.21.15.7:0/939905829 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78007dc0 con 0x7fbe801e09e0 2024-03-19T01:23:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe67fff640 1 -- 172.21.15.7:0/939905829 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe78001260 con 0x7fbe801e09e0 2024-03-19T01:23:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe67fff640 1 -- 172.21.15.7:0/939905829 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78006d70 con 0x7fbe801e09e0 2024-03-19T01:23:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe800ae2d0 con 0x7fbe801e09e0 2024-03-19T01:23:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe800adf30 con 0x7fbe801e09e0 2024-03-19T01:23:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.747+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe800ae4a0 con 0x7fbe801e09e0 2024-03-19T01:23:36.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.751+0000 7fbe67fff640 1 -- 172.21.15.7:0/939905829 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbe780a6480 con 0x7fbe801e09e0 2024-03-19T01:23:36.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.751+0000 7fbe67fff640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbe6c067660 0x7fbe6c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:36.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.751+0000 7fbe67fff640 1 -- 172.21.15.7:0/939905829 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbe7811f3d0 con 0x7fbe801e09e0 2024-03-19T01:23:36.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.751+0000 7fbe7f7fe640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbe6c067660 0x7fbe6c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:36.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.751+0000 7fbe7f7fe640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbe6c067660 0x7fbe6c069b20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbe700098f0 tx=0x7fbe70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:36.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.759+0000 7fbe67fff640 1 -- 172.21.15.7:0/939905829 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbe800ae4a0 con 0x7fbe801e09e0 2024-03-19T01:23:36.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fbe801dfd70 con 0x7fbe801e09e0 2024-03-19T01:23:36.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe67fff640 1 -- 172.21.15.7:0/939905829 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fbe78016490 con 0x7fbe801e09e0 2024-03-19T01:23:36.952 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:36.953 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbe6c067660 msgr2=0x7fbe6c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbe6c067660 0x7fbe6c069b20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbe700098f0 tx=0x7fbe70008040 comp rx=0 tx=0).stop 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe801e09e0 msgr2=0x7fbe80138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe801e09e0 0x7fbe80138d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbe7801c5b0 tx=0x7fbe68005840 comp rx=0 tx=0).stop 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 shutdown_connections 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbe6c067660 0x7fbe6c069b20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe800ae780 0x7fbe801e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe801e09e0 0x7fbe80138d20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 --2- 172.21.15.7:0/939905829 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe680051d0 0x7fbe800af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 >> 172.21.15.7:0/939905829 conn(0x7fbe800b68e0 msgr2=0x7fbe800b6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 shutdown_connections 2024-03-19T01:23:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:36.947+0000 7fbe86250640 1 -- 172.21.15.7:0/939905829 wait complete. 2024-03-19T01:23:36.969 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-19T01:23:36.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.079+0000 7fad141fe640 1 Processor -- start 2024-03-19T01:23:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.079+0000 7fad141fe640 1 -- start start 2024-03-19T01:23:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad141fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fad0c0ae190 0x7fad0c0b5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad141fe640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0c0b34c0 con 0x7fad0c14fd80 2024-03-19T01:23:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad141fe640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0c0adac0 con 0x7fad0c14b500 2024-03-19T01:23:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad141fe640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fad0c0ad870 con 0x7fad0c0ae190 2024-03-19T01:23:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad1279b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fad0c14fd80 0x7fad0c152170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43278/0 (socket says 172.21.15.7:43278) 2024-03-19T01:23:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad1279b640 1 -- 172.21.15.7:0/1136849083 learned_addr learned my addr 172.21.15.7:0/1136849083 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11799640 1 --2- 172.21.15.7:0/1136849083 >> v2:172.21.15.167:3300/0 conn(0x7fad0c0ae190 0x7fad0c0b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3181582020 0 0) 0x7fad0c0b34c0 con 0x7fad0c14fd80 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facf8003530 con 0x7fad0c14fd80 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1698974382 0 0) 0x7facf8003530 con 0x7fad0c14fd80 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 >> v2:172.21.15.167:3300/0 conn(0x7fad0c0ae190 msgr2=0x7fad0c0b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 --2- 172.21.15.7:0/1136849083 >> v2:172.21.15.167:3300/0 conn(0x7fad0c0ae190 0x7fad0c0b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 >> v1:172.21.15.167:6789/0 conn(0x7fad0c14b500 legacy=0x7fad0c074130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c0ae900 con 0x7fad0c14fd80 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fad04001a00 con 0x7fad0c14fd80 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 >> v1:172.21.15.7:6789/0 conn(0x7fad0c14fd80 legacy=0x7fad0c152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facf80039c0 0x7facf8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf8004810 0x7facf8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7facf80051d0 0x7facf800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c0ad870 con 0x7facf80039c0 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7facf800c170 con 0x7facf8004810 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7facf800c370 con 0x7facf80051d0 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11f9a640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facf80039c0 0x7facf8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad1279b640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7facf80051d0 0x7facf800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11799640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf8004810 0x7facf8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11f9a640 1 -- 172.21.15.7:0/1136849083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7facf80051d0 msgr2=0x7facf800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11f9a640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7facf80051d0 0x7facf800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11f9a640 1 -- 172.21.15.7:0/1136849083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf8004810 msgr2=0x7facf8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11f9a640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf8004810 0x7facf8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11f9a640 1 -- 172.21.15.7:0/1136849083 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf8003530 con 0x7facf80039c0 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad1279b640 1 -- 172.21.15.7:0/1136849083 reap_dead start 2024-03-19T01:23:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11799640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf8004810 0x7facf8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad11f9a640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facf80039c0 0x7facf8003de0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad080017b0 tx=0x7fad0801c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa05b63211f66c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.083+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad080013d0 con 0x7facf80039c0 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad080052d0 con 0x7facf80039c0 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad10f98640 1 -- 172.21.15.7:0/1136849083 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad08003650 con 0x7facf80039c0 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- 172.21.15.7:0/1136849083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facf80039c0 msgr2=0x7facf8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facf80039c0 0x7facf8003de0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad080017b0 tx=0x7fad0801c390 comp rx=0 tx=0).stop 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- 172.21.15.7:0/1136849083 shutdown_connections 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 --2- 172.21.15.7:0/1136849083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facf80039c0 0x7facf8003de0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- 172.21.15.7:0/1136849083 >> 172.21.15.7:0/1136849083 conn(0x7fad0c0b5c20 msgr2=0x7fad0c0b6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- 172.21.15.7:0/1136849083 shutdown_connections 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- 172.21.15.7:0/1136849083 wait complete. 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 Processor -- start 2024-03-19T01:23:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- start start 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf80039c0 0x7fad0c149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad0c1dfe30 0x7fad0c1437a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad0c143ce0 0x7fad0c1e1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c0b5590 con 0x7fad0c143ce0 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c0b5410 con 0x7facf80039c0 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c0b5710 con 0x7fad0c1dfe30 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad1279b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad0c143ce0 0x7fad0c1e1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad1279b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad0c143ce0 0x7fad0c1e1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40250/0 (socket says 172.21.15.7:40250) 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad1279b640 1 -- 172.21.15.7:0/2964659388 learned_addr learned my addr 172.21.15.7:0/2964659388 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad11f9a640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf80039c0 0x7fad0c149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad11799640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad0c1dfe30 0x7fad0c1437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad1279b640 1 -- 172.21.15.7:0/2964659388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad0c1dfe30 msgr2=0x7fad0c1437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad1279b640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad0c1dfe30 0x7fad0c1437a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad1279b640 1 -- 172.21.15.7:0/2964659388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf80039c0 msgr2=0x7fad0c149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad1279b640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf80039c0 0x7fad0c149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad1279b640 1 -- 172.21.15.7:0/2964659388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c1e0330 con 0x7fad0c143ce0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad1279b640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad0c143ce0 0x7fad0c1e1320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0b5b20 tx=0x7fad04007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad027fc640 1 -- 172.21.15.7:0/2964659388 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04017020 con 0x7fad0c143ce0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad027fc640 1 -- 172.21.15.7:0/2964659388 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0400ab00 con 0x7fad0c143ce0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0c1e0610 con 0x7fad0c143ce0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad027fc640 1 -- 172.21.15.7:0/2964659388 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad040133f0 con 0x7fad0c143ce0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.087+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0c14a010 con 0x7fad0c143ce0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.091+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad0c1e0df0 con 0x7fad0c143ce0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.091+0000 7fad027fc640 1 -- 172.21.15.7:0/2964659388 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fad0401a020 con 0x7fad0c143ce0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.091+0000 7fad027fc640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7facf0067690 0x7facf0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.091+0000 7fad11f9a640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7facf0067690 0x7facf0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.091+0000 7fad027fc640 1 -- 172.21.15.7:0/2964659388 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fad04085ef0 con 0x7fad0c143ce0 2024-03-19T01:23:37.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.091+0000 7fad11f9a640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7facf0067690 0x7facf0069b50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad080ac000 tx=0x7fad080011f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:37.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.095+0000 7fad027fc640 1 -- 172.21.15.7:0/2964659388 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fad0c1e0df0 con 0x7fad0c143ce0 2024-03-19T01:23:37.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.259+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad0c075460 con 0x7fad0c143ce0 2024-03-19T01:23:37.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad027fc640 1 -- 172.21.15.7:0/2964659388 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7fad04021020 con 0x7fad0c143ce0 2024-03-19T01:23:37.268 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:37.268 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:37.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7facf0067690 msgr2=0x7facf0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7facf0067690 0x7facf0069b50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad080ac000 tx=0x7fad080011f0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad0c143ce0 msgr2=0x7fad0c1e1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad0c143ce0 0x7fad0c1e1320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0b5b20 tx=0x7fad04007b00 comp rx=0 tx=0).stop 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 shutdown_connections 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7facf0067690 0x7facf0069b50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad0c1dfe30 0x7fad0c1437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad0c143ce0 0x7fad0c1e1320 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 --2- 172.21.15.7:0/2964659388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7facf80039c0 0x7fad0c149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 >> 172.21.15.7:0/2964659388 conn(0x7fad0c0b5c20 msgr2=0x7fad0c144450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 shutdown_connections 2024-03-19T01:23:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.263+0000 7fad141fe640 1 -- 172.21.15.7:0/2964659388 wait complete. 2024-03-19T01:23:37.286 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:37.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc713ce640 1 Processor -- start 2024-03-19T01:23:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc713ce640 1 -- start start 2024-03-19T01:23:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc713ce640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcc6c145600 0x7fcc6c144bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc713ce640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fcc6c13e040 0x7fcc6c13e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc713ce640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc6c144110 con 0x7fcc6c067260 2024-03-19T01:23:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc713ce640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fcc6c1459e0 con 0x7fcc6c145600 2024-03-19T01:23:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc713ce640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fcc6c0730b0 con 0x7fcc6c13e040 2024-03-19T01:23:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc63fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcc6c145600 0x7fcc6c144bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc6affd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcc6c067260 0x7fcc6c0b2900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43290/0 (socket says 172.21.15.7:43290) 2024-03-19T01:23:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc6affd640 1 -- 172.21.15.7:0/2329768756 learned_addr learned my addr 172.21.15.7:0/2329768756 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc6b7fe640 1 --2- 172.21.15.7:0/2329768756 >> v2:172.21.15.167:3300/0 conn(0x7fcc6c13e040 0x7fcc6c13e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc63fff640 1 -- 172.21.15.7:0/2329768756 >> v2:172.21.15.167:3300/0 conn(0x7fcc6c13e040 msgr2=0x7fcc6c13e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc63fff640 1 --2- 172.21.15.7:0/2329768756 >> v2:172.21.15.167:3300/0 conn(0x7fcc6c13e040 0x7fcc6c13e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc63fff640 1 -- 172.21.15.7:0/2329768756 >> v1:172.21.15.7:6789/0 conn(0x7fcc6c067260 legacy=0x7fcc6c0b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc63fff640 1 -- 172.21.15.7:0/2329768756 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6c067640 con 0x7fcc6c145600 2024-03-19T01:23:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc6a7fc640 1 -- 172.21.15.7:0/2329768756 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2545723896 0 0) 0x7fcc6c144110 con 0x7fcc6c067260 2024-03-19T01:23:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc63fff640 1 --2- 172.21.15.7:0/2329768756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 0x7fcc6c144bb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c147630 tx=0x7fcc5401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eea3f69ee2d184 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc6a7fc640 1 -- 172.21.15.7:0/2329768756 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc540b0070 con 0x7fcc6c145600 2024-03-19T01:23:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc6a7fc640 1 -- 172.21.15.7:0/2329768756 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc540ab600 con 0x7fcc6c145600 2024-03-19T01:23:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.403+0000 7fcc6a7fc640 1 -- 172.21.15.7:0/2329768756 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc540ab9d0 con 0x7fcc6c145600 2024-03-19T01:23:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- 172.21.15.7:0/2329768756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 msgr2=0x7fcc6c144bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 --2- 172.21.15.7:0/2329768756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 0x7fcc6c144bb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c147630 tx=0x7fcc5401c480 comp rx=0 tx=0).stop 2024-03-19T01:23:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- 172.21.15.7:0/2329768756 shutdown_connections 2024-03-19T01:23:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 --2- 172.21.15.7:0/2329768756 >> v2:172.21.15.167:3300/0 conn(0x7fcc6c13e040 0x7fcc6c13e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 --2- 172.21.15.7:0/2329768756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 0x7fcc6c144bb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- 172.21.15.7:0/2329768756 >> 172.21.15.7:0/2329768756 conn(0x7fcc6c0b9290 msgr2=0x7fcc6c0b9670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- 172.21.15.7:0/2329768756 shutdown_connections 2024-03-19T01:23:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- 172.21.15.7:0/2329768756 wait complete. 2024-03-19T01:23:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 Processor -- start 2024-03-19T01:23:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- start start 2024-03-19T01:23:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc6c067260 0x7fcc6c142980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc6c13e040 0x7fcc6c0b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 0x7fcc6c140aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6c1514c0 con 0x7fcc6c145600 2024-03-19T01:23:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6c0b30c0 con 0x7fcc6c067260 2024-03-19T01:23:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcc6c151640 con 0x7fcc6c13e040 2024-03-19T01:23:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 0x7fcc6c140aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 0x7fcc6c140aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40274/0 (socket says 172.21.15.7:40274) 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6b7fe640 1 -- 172.21.15.7:0/2112414850 learned_addr learned my addr 172.21.15.7:0/2112414850 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc63fff640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc6c13e040 0x7fcc6c0b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6affd640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc6c067260 0x7fcc6c142980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6b7fe640 1 -- 172.21.15.7:0/2112414850 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc6c13e040 msgr2=0x7fcc6c0b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6b7fe640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc6c13e040 0x7fcc6c0b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6b7fe640 1 -- 172.21.15.7:0/2112414850 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc6c067260 msgr2=0x7fcc6c142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6b7fe640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc6c067260 0x7fcc6c142980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6b7fe640 1 -- 172.21.15.7:0/2112414850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6c1ddc90 con 0x7fcc6c145600 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6affd640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc6c067260 0x7fcc6c142980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc6b7fe640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 0x7fcc6c140aa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcc64007e40 tx=0x7fcc6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc637fe640 1 -- 172.21.15.7:0/2112414850 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64014620 con 0x7fcc6c145600 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc637fe640 1 -- 172.21.15.7:0/2112414850 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc64019070 con 0x7fcc6c145600 2024-03-19T01:23:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc637fe640 1 -- 172.21.15.7:0/2112414850 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64014a20 con 0x7fcc6c145600 2024-03-19T01:23:37.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc6c1de1f0 con 0x7fcc6c145600 2024-03-19T01:23:37.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc6c139cc0 con 0x7fcc6c145600 2024-03-19T01:23:37.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.407+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc6c075570 con 0x7fcc6c145600 2024-03-19T01:23:37.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.411+0000 7fcc637fe640 1 -- 172.21.15.7:0/2112414850 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcc64014bc0 con 0x7fcc6c145600 2024-03-19T01:23:37.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.411+0000 7fcc637fe640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc4c067690 0x7fcc4c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.411+0000 7fcc637fe640 1 -- 172.21.15.7:0/2112414850 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcc64086fc0 con 0x7fcc6c145600 2024-03-19T01:23:37.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.411+0000 7fcc6affd640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc4c067690 0x7fcc4c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.411+0000 7fcc6affd640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc4c067690 0x7fcc4c069b50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c000f70 tx=0x7fcc5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:37.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.415+0000 7fcc637fe640 1 -- 172.21.15.7:0/2112414850 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcc6c1ddc90 con 0x7fcc6c145600 2024-03-19T01:23:37.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.567+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcc6c1de6a0 con 0x7fcc6c145600 2024-03-19T01:23:37.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.567+0000 7fcc637fe640 1 -- 172.21.15.7:0/2112414850 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fcc64057d20 con 0x7fcc6c145600 2024-03-19T01:23:37.574 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:37.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc4c067690 msgr2=0x7fcc4c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc4c067690 0x7fcc4c069b50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c000f70 tx=0x7fcc5c009040 comp rx=0 tx=0).stop 2024-03-19T01:23:37.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 msgr2=0x7fcc6c140aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 0x7fcc6c140aa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcc64007e40 tx=0x7fcc6400e4b0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 shutdown_connections 2024-03-19T01:23:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc4c067690 0x7fcc4c069b50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc6c13e040 0x7fcc6c0b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc6c145600 0x7fcc6c140aa0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 --2- 172.21.15.7:0/2112414850 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc6c067260 0x7fcc6c142980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 >> 172.21.15.7:0/2112414850 conn(0x7fcc6c0b9290 msgr2=0x7fcc6c070680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 shutdown_connections 2024-03-19T01:23:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.571+0000 7fcc713ce640 1 -- 172.21.15.7:0/2112414850 wait complete. 2024-03-19T01:23:37.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:37.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0117ca0640 1 Processor -- start 2024-03-19T01:23:37.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0117ca0640 1 -- start start 2024-03-19T01:23:37.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0117ca0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0110074390 0x7f011013f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0117ca0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f011013f730 0x7f0110145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0117ca0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0110074770 con 0x7f01101460f0 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0117ca0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f0110073730 con 0x7f0110074390 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0117ca0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0110145720 con 0x7f011013f730 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0115a3c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0110074390 0x7f011013f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0115a3c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0110074390 0x7f011013f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40284/0 (socket says 172.21.15.7:40284) 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0115a3c640 1 -- 172.21.15.7:0/3700038249 learned_addr learned my addr 172.21.15.7:0/3700038249 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f011523b640 1 --2- 172.21.15.7:0/3700038249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f011013f730 0x7f0110145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0115a3c640 1 -- 172.21.15.7:0/3700038249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f011013f730 msgr2=0x7f0110145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0115a3c640 1 --2- 172.21.15.7:0/3700038249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f011013f730 0x7f0110145bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0115a3c640 1 -- 172.21.15.7:0/3700038249 >> v1:172.21.15.167:6789/0 conn(0x7f01101460f0 legacy=0x7f0110148a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0115a3c640 1 -- 172.21.15.7:0/3700038249 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01100b3c00 con 0x7f0110074390 2024-03-19T01:23:37.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0115a3c640 1 --2- 172.21.15.7:0/3700038249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0110074390 0x7f011013f1f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f010c009870 tx=0x7f010c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5839812aede23d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:37.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.711+0000 7f0114a3a640 1 -- 172.21.15.7:0/3700038249 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c0b0070 con 0x7f0110074390 2024-03-19T01:23:37.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0114a3a640 1 -- 172.21.15.7:0/3700038249 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f010c0ab600 con 0x7f0110074390 2024-03-19T01:23:37.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0114a3a640 1 -- 172.21.15.7:0/3700038249 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c0ab9d0 con 0x7f0110074390 2024-03-19T01:23:37.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- 172.21.15.7:0/3700038249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0110074390 msgr2=0x7f011013f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 --2- 172.21.15.7:0/3700038249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0110074390 0x7f011013f1f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f010c009870 tx=0x7f010c01c4d0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- 172.21.15.7:0/3700038249 shutdown_connections 2024-03-19T01:23:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 --2- 172.21.15.7:0/3700038249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f011013f730 0x7f0110145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 --2- 172.21.15.7:0/3700038249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0110074390 0x7f011013f1f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- 172.21.15.7:0/3700038249 >> 172.21.15.7:0/3700038249 conn(0x7f011006bc70 msgr2=0x7f01100b9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- 172.21.15.7:0/3700038249 shutdown_connections 2024-03-19T01:23:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- 172.21.15.7:0/3700038249 wait complete. 2024-03-19T01:23:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 Processor -- start 2024-03-19T01:23:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- start start 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0110074390 0x7f01101421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f011013f730 0x7f011013dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01101460f0 0x7f01101e3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0110152fa0 con 0x7f011013f730 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0110152e20 con 0x7f0110074390 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0110153120 con 0x7f01101460f0 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011523b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f011013f730 0x7f011013dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011523b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f011013f730 0x7f011013dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40296/0 (socket says 172.21.15.7:40296) 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011523b640 1 -- 172.21.15.7:0/3631646259 learned_addr learned my addr 172.21.15.7:0/3631646259 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0115a3c640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0110074390 0x7f01101421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011623d640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01101460f0 0x7f01101e3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011523b640 1 -- 172.21.15.7:0/3631646259 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01101460f0 msgr2=0x7f01101e3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011523b640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01101460f0 0x7f01101e3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011523b640 1 -- 172.21.15.7:0/3631646259 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0110074390 msgr2=0x7f01101421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011523b640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0110074390 0x7f01101421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011523b640 1 -- 172.21.15.7:0/3631646259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0110142a10 con 0x7f011013f730 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011623d640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01101460f0 0x7f01101e3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f011523b640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f011013f730 0x7f011013dfb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f010000d7d0 tx=0x7f010000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f01067fc640 1 -- 172.21.15.7:0/3631646259 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010000b660 con 0x7f011013f730 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f01067fc640 1 -- 172.21.15.7:0/3631646259 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0100019070 con 0x7f011013f730 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01101e4300 con 0x7f011013f730 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f01067fc640 1 -- 172.21.15.7:0/3631646259 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010000ba40 con 0x7f011013f730 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.715+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0110142720 con 0x7f011013f730 2024-03-19T01:23:37.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.719+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01100b8b50 con 0x7f011013f730 2024-03-19T01:23:37.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.719+0000 7f01067fc640 1 -- 172.21.15.7:0/3631646259 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f010000bbe0 con 0x7f011013f730 2024-03-19T01:23:37.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.719+0000 7f01067fc640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f00f40676c0 0x7f00f4069b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:37.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.719+0000 7f01067fc640 1 -- 172.21.15.7:0/3631646259 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0100085c10 con 0x7f011013f730 2024-03-19T01:23:37.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.719+0000 7f0115a3c640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f00f40676c0 0x7f00f4069b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:37.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.719+0000 7f0115a3c640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f00f40676c0 0x7f00f4069b80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f010c0ad000 tx=0x7f010c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:37.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.723+0000 7f01067fc640 1 -- 172.21.15.7:0/3631646259 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0100056920 con 0x7f011013f730 2024-03-19T01:23:37.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.887+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f011013eda0 con 0x7f011013f730 2024-03-19T01:23:37.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.887+0000 7f01067fc640 1 -- 172.21.15.7:0/3631646259 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f0100056220 con 0x7f011013f730 2024-03-19T01:23:37.893 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f00f40676c0 msgr2=0x7f00f4069b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f00f40676c0 0x7f00f4069b80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f010c0ad000 tx=0x7f010c0a6000 comp rx=0 tx=0).stop 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f011013f730 msgr2=0x7f011013dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f011013f730 0x7f011013dfb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f010000d7d0 tx=0x7f010000dc90 comp rx=0 tx=0).stop 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 shutdown_connections 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f00f40676c0 0x7f00f4069b80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01101460f0 0x7f01101e3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f011013f730 0x7f011013dfb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 --2- 172.21.15.7:0/3631646259 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0110074390 0x7f01101421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 >> 172.21.15.7:0/3631646259 conn(0x7f011006bc70 msgr2=0x7f01100b63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 shutdown_connections 2024-03-19T01:23:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:37.891+0000 7f0117ca0640 1 -- 172.21.15.7:0/3631646259 wait complete. 2024-03-19T01:23:37.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:38.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.019+0000 7f0fddae3640 1 Processor -- start 2024-03-19T01:23:38.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.019+0000 7f0fddae3640 1 -- start start 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fddae3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fddae3640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0fd80b69f0 0x7f0fd80b4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fddae3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd80b2c20 con 0x7f0fd80b6e50 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fddae3640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f0fd80b5d00 con 0x7f0fd80b69f0 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fddae3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd80b5fe0 con 0x7f0fd80b77f0 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd77fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd6ffd640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0fd80b69f0 0x7f0fd80b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd77fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:55382/0 (socket says 172.21.15.7:55382) 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd77fe640 1 -- 172.21.15.7:0/2584563154 learned_addr learned my addr 172.21.15.7:0/2584563154 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd67fc640 1 -- 172.21.15.7:0/2584563154 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2693666405 0 0) 0x7f0fd80b2c20 con 0x7f0fd80b6e50 2024-03-19T01:23:38.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd67fc640 1 -- 172.21.15.7:0/2584563154 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fc0003530 con 0x7f0fd80b6e50 2024-03-19T01:23:38.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd77fe640 1 -- 172.21.15.7:0/2584563154 >> v2:172.21.15.167:3300/0 conn(0x7f0fd80b69f0 msgr2=0x7f0fd80b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:38.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd77fe640 1 --2- 172.21.15.7:0/2584563154 >> v2:172.21.15.167:3300/0 conn(0x7f0fd80b69f0 0x7f0fd80b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd77fe640 1 -- 172.21.15.7:0/2584563154 >> v1:172.21.15.167:6789/0 conn(0x7f0fd80b6e50 legacy=0x7f0fd80714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.023+0000 7f0fd77fe640 1 -- 172.21.15.7:0/2584563154 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd80b64d0 con 0x7f0fd80b77f0 2024-03-19T01:23:38.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 --2- 172.21.15.7:0/2584563154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b56e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc009870 tx=0x7f0fcc01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bde8739cd85d0f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd67fc640 1 -- 172.21.15.7:0/2584563154 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc0aa650 con 0x7f0fd80b77f0 2024-03-19T01:23:38.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd67fc640 1 -- 172.21.15.7:0/2584563154 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fcc0af070 con 0x7f0fd80b77f0 2024-03-19T01:23:38.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd67fc640 1 -- 172.21.15.7:0/2584563154 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc0aa9f0 con 0x7f0fd80b77f0 2024-03-19T01:23:38.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 -- 172.21.15.7:0/2584563154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 msgr2=0x7f0fd80b56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 --2- 172.21.15.7:0/2584563154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b56e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc009870 tx=0x7f0fcc01c4d0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 -- 172.21.15.7:0/2584563154 shutdown_connections 2024-03-19T01:23:38.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 --2- 172.21.15.7:0/2584563154 >> v2:172.21.15.167:3300/0 conn(0x7f0fd80b69f0 0x7f0fd80b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 --2- 172.21.15.7:0/2584563154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b56e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 -- 172.21.15.7:0/2584563154 >> 172.21.15.7:0/2584563154 conn(0x7f0fd806b2c0 msgr2=0x7f0fd806b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:38.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 -- 172.21.15.7:0/2584563154 shutdown_connections 2024-03-19T01:23:38.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 -- 172.21.15.7:0/2584563154 wait complete. 2024-03-19T01:23:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 Processor -- start 2024-03-19T01:23:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 -- start start 2024-03-19T01:23:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd80b69f0 0x7f0fd8145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fd80b6e50 0x7f0fd80b8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd81489b0 con 0x7f0fd80b69f0 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd8148830 con 0x7f0fd80b6e50 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fddae3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd8152b80 con 0x7f0fd80b77f0 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd80b69f0 0x7f0fd8145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd80b69f0 0x7f0fd8145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40312/0 (socket says 172.21.15.7:40312) 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 -- 172.21.15.7:0/2179956946 learned_addr learned my addr 172.21.15.7:0/2179956946 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd7fff640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd6ffd640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fd80b6e50 0x7f0fd80b8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 -- 172.21.15.7:0/2179956946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 msgr2=0x7f0fd80b94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 -- 172.21.15.7:0/2179956946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fd80b6e50 msgr2=0x7f0fd80b8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fd80b6e50 0x7f0fd80b8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 -- 172.21.15.7:0/2179956946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd8141c60 con 0x7f0fd80b69f0 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd6ffd640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fd80b6e50 0x7f0fd80b8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fd77fe640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd80b69f0 0x7f0fd8145660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0ac040 tx=0x7f0fcc019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.027+0000 7f0fbffff640 1 -- 172.21.15.7:0/2179956946 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc003b70 con 0x7f0fd80b69f0 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fbffff640 1 -- 172.21.15.7:0/2179956946 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fcc0a6040 con 0x7f0fd80b69f0 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fbffff640 1 -- 172.21.15.7:0/2179956946 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc0aa450 con 0x7f0fd80b69f0 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd80bb4e0 con 0x7f0fd80b69f0 2024-03-19T01:23:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fd813cbd0 con 0x7f0fd80b69f0 2024-03-19T01:23:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd80bb2f0 con 0x7f0fd80b69f0 2024-03-19T01:23:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fbffff640 1 -- 172.21.15.7:0/2179956946 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0fcc0aa5f0 con 0x7f0fd80b69f0 2024-03-19T01:23:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fbffff640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0fc4067660 0x7f0fc4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fbffff640 1 -- 172.21.15.7:0/2179956946 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0fcc11ff00 con 0x7f0fd80b69f0 2024-03-19T01:23:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fd6ffd640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0fc4067660 0x7f0fc4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.031+0000 7f0fd6ffd640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0fc4067660 0x7f0fc4069b20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8149160 tx=0x7f0fc8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.035+0000 7f0fbffff640 1 -- 172.21.15.7:0/2179956946 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0fcc0b8030 con 0x7f0fd80b69f0 2024-03-19T01:23:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.191+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0fd8075570 con 0x7f0fd80b69f0 2024-03-19T01:23:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.191+0000 7f0fbffff640 1 -- 172.21.15.7:0/2179956946 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f0fcc016150 con 0x7f0fd80b69f0 2024-03-19T01:23:38.198 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0fc4067660 msgr2=0x7f0fc4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0fc4067660 0x7f0fc4069b20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8149160 tx=0x7f0fc8008040 comp rx=0 tx=0).stop 2024-03-19T01:23:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd80b69f0 msgr2=0x7f0fd8145660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd80b69f0 0x7f0fd8145660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0ac040 tx=0x7f0fcc019270 comp rx=0 tx=0).stop 2024-03-19T01:23:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 shutdown_connections 2024-03-19T01:23:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0fc4067660 0x7f0fc4069b20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fd80b77f0 0x7f0fd80b94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fd80b6e50 0x7f0fd80b8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 --2- 172.21.15.7:0/2179956946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd80b69f0 0x7f0fd8145660 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 >> 172.21.15.7:0/2179956946 conn(0x7f0fd806b2c0 msgr2=0x7f0fd813b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:38.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 shutdown_connections 2024-03-19T01:23:38.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.195+0000 7f0fddae3640 1 -- 172.21.15.7:0/2179956946 wait complete. 2024-03-19T01:23:38.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35f1ea2640 1 Processor -- start 2024-03-19T01:23:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35f1ea2640 1 -- start start 2024-03-19T01:23:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35f1ea2640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f35ec0b3030 0x7f35ec0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35f1ea2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35ec070540 con 0x7f35ec1479a0 2024-03-19T01:23:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35f1ea2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35ec148490 con 0x7f35ec070d40 2024-03-19T01:23:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35f1ea2640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f35ec0702f0 con 0x7f35ec0b3030 2024-03-19T01:23:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eb7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f35ec1479a0 0x7f35ec147d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43298/0 (socket says 172.21.15.7:43298) 2024-03-19T01:23:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eb7fe640 1 -- 172.21.15.7:0/1655685042 learned_addr learned my addr 172.21.15.7:0/1655685042 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eaffd640 1 --2- 172.21.15.7:0/1655685042 >> v2:172.21.15.167:3300/0 conn(0x7f35ec0b3030 0x7f35ec0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1822314437 0 0) 0x7f35ec070540 con 0x7f35ec1479a0 2024-03-19T01:23:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35d4005740 con 0x7f35ec1479a0 2024-03-19T01:23:38.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2844021 0 0) 0x7f35d4005740 con 0x7f35ec1479a0 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 >> v2:172.21.15.167:3300/0 conn(0x7f35ec0b3030 msgr2=0x7f35ec0b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 --2- 172.21.15.7:0/1655685042 >> v2:172.21.15.167:3300/0 conn(0x7f35ec0b3030 0x7f35ec0b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 >> v1:172.21.15.167:6789/0 conn(0x7f35ec070d40 legacy=0x7f35ec071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35ec0b9810 con 0x7f35ec1479a0 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f35dc001680 con 0x7f35ec1479a0 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 >> v1:172.21.15.7:6789/0 conn(0x7f35ec1479a0 legacy=0x7f35ec147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35d4005bd0 0x7f35d4005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35d4006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35d40073e0 0x7f35d400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f35ec0702f0 con 0x7f35d4006a20 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f35d400e380 con 0x7f35d40073e0 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f35d400e580 con 0x7f35d4005bd0 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eaffd640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35d4006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eb7fe640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35d4005bd0 0x7f35d4005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ebfff640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35d40073e0 0x7f35d400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eaffd640 1 -- 172.21.15.7:0/1655685042 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35d4005bd0 msgr2=0x7f35d4005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eaffd640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35d4005bd0 0x7f35d4005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eaffd640 1 -- 172.21.15.7:0/1655685042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35d40073e0 msgr2=0x7f35d400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eaffd640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35d40073e0 0x7f35d400dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35eaffd640 1 -- 172.21.15.7:0/1655685042 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d4005740 con 0x7f35d4006a20 2024-03-19T01:23:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ebfff640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35d40073e0 0x7f35d400dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.323+0000 7f35ebfff640 1 -- 172.21.15.7:0/1655685042 reap_dead start 2024-03-19T01:23:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35eaffd640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35d4006ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009ae0 tx=0x7f35e001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa510c9dd4efcf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e00096b0 con 0x7f35d4006a20 2024-03-19T01:23:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e00a6450 con 0x7f35d4006a20 2024-03-19T01:23:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35ea7fc640 1 -- 172.21.15.7:0/1655685042 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e00a6840 con 0x7f35d4006a20 2024-03-19T01:23:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 -- 172.21.15.7:0/1655685042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 msgr2=0x7f35d4006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35d4006ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009ae0 tx=0x7f35e001ca70 comp rx=0 tx=0).stop 2024-03-19T01:23:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 -- 172.21.15.7:0/1655685042 shutdown_connections 2024-03-19T01:23:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 --2- 172.21.15.7:0/1655685042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35d4006ea0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 -- 172.21.15.7:0/1655685042 >> 172.21.15.7:0/1655685042 conn(0x7f35ec0b7a10 msgr2=0x7f35ec0b7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:38.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 -- 172.21.15.7:0/1655685042 shutdown_connections 2024-03-19T01:23:38.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 -- 172.21.15.7:0/1655685042 wait complete. 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 Processor -- start 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 -- start start 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35ec146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35ec1472a0 0x7f35ec1dcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35ec136d20 0x7f35ec1371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f35ec0714f0 con 0x7f35d4006a20 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f35ec071370 con 0x7f35ec136d20 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.327+0000 7f35f1ea2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f35ec071670 con 0x7f35ec1472a0 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35ec146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35ec146210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40330/0 (socket says 172.21.15.7:40330) 2024-03-19T01:23:38.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eb7fe640 1 -- 172.21.15.7:0/649073885 learned_addr learned my addr 172.21.15.7:0/649073885 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eaffd640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35ec1472a0 0x7f35ec1dcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35ebfff640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35ec136d20 0x7f35ec1371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eb7fe640 1 -- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35ec1472a0 msgr2=0x7f35ec1dcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eb7fe640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35ec1472a0 0x7f35ec1dcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eb7fe640 1 -- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35ec136d20 msgr2=0x7f35ec1371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eb7fe640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35ec136d20 0x7f35ec1371a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eb7fe640 1 -- 172.21.15.7:0/649073885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35ec1376e0 con 0x7f35d4006a20 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eaffd640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35ec1472a0 0x7f35ec1dcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35ebfff640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35ec136d20 0x7f35ec1371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eb7fe640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35ec146210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f35dc00e870 tx=0x7f35dc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35d3fff640 1 -- 172.21.15.7:0/649073885 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc00c870 con 0x7f35d4006a20 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35d3fff640 1 -- 172.21.15.7:0/649073885 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35dc018070 con 0x7f35d4006a20 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35ec1dc3e0 con 0x7f35d4006a20 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35d3fff640 1 -- 172.21.15.7:0/649073885 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc00cc30 con 0x7f35d4006a20 2024-03-19T01:23:38.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35ec1dd120 con 0x7f35d4006a20 2024-03-19T01:23:38.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35ec075460 con 0x7f35d4006a20 2024-03-19T01:23:38.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35d3fff640 1 -- 172.21.15.7:0/649073885 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f35dc026430 con 0x7f35d4006a20 2024-03-19T01:23:38.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35d3fff640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35d8067690 0x7f35d8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eaffd640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35d8067690 0x7f35d8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35d3fff640 1 -- 172.21.15.7:0/649073885 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f35dc085f90 con 0x7f35d4006a20 2024-03-19T01:23:38.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.331+0000 7f35eaffd640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35d8067690 0x7f35d8069b50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f35e00ac000 tx=0x7f35e0004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.335+0000 7f35d3fff640 1 -- 172.21.15.7:0/649073885 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f35ec1376e0 con 0x7f35d4006a20 2024-03-19T01:23:38.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.527+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35ec141ae0 con 0x7f35d4006a20 2024-03-19T01:23:38.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.527+0000 7f35d3fff640 1 -- 172.21.15.7:0/649073885 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3524 (secure 0 0 0) 0x7f35dc056bf0 con 0x7f35d4006a20 2024-03-19T01:23:38.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:38.535 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4208,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/656985991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":656985991},{"type":"v1","addr":"172.21.15.167:6833","nonce":656985991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:23:26.868603+0000","modified":"2024-03-19T01:23:29.918429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-03-19T01:23:38.535 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-03-19T01:23:38.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35d8067690 msgr2=0x7f35d8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35d8067690 0x7f35d8069b50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f35e00ac000 tx=0x7f35e0004740 comp rx=0 tx=0).stop 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 msgr2=0x7f35ec146210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35ec146210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f35dc00e870 tx=0x7f35dc00ed30 comp rx=0 tx=0).stop 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 shutdown_connections 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35d8067690 0x7f35d8069b50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35ec1472a0 0x7f35ec1dcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35ec136d20 0x7f35ec1371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 --2- 172.21.15.7:0/649073885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d4006a20 0x7f35ec146210 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 >> 172.21.15.7:0/649073885 conn(0x7f35ec0b7a10 msgr2=0x7f35ec06ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 shutdown_connections 2024-03-19T01:23:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.531+0000 7f35f1ea2640 1 -- 172.21.15.7:0/649073885 wait complete. 2024-03-19T01:23:38.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:38.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.651+0000 7f1f58bef640 1 Processor -- start 2024-03-19T01:23:38.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.651+0000 7f1f58bef640 1 -- start start 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f58bef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f54074390 0x7f1f54141430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f58bef640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f540aead0 con 0x7f1f54141970 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f58bef640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f54074770 con 0x7f1f540add50 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f58bef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1f540af050 con 0x7f1f54074390 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f52d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1f54141970 0x7f1f54151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43304/0 (socket says 172.21.15.7:43304) 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f52d9d640 1 -- 172.21.15.7:0/3640009554 learned_addr learned my addr 172.21.15.7:0/3640009554 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5259c640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f54074390 0x7f1f54141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457689166 0 0) 0x7f1f540aead0 con 0x7f1f54141970 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f3c003530 con 0x7f1f54141970 2024-03-19T01:23:38.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2304832574 0 0) 0x7f1f3c003530 con 0x7f1f54141970 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f54074390 msgr2=0x7f1f54141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f54074390 0x7f1f54141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 >> v1:172.21.15.167:6789/0 conn(0x7f1f540add50 legacy=0x7f1f54147830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f540aed70 con 0x7f1f54141970 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f1f4c001a20 con 0x7f1f54141970 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 >> v1:172.21.15.7:6789/0 conn(0x7f1f54141970 legacy=0x7f1f54151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f3c0039f0 0x7f1f3c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c004810 0x7f1f3c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f54074390 0x7f1f3c0052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1f540af050 con 0x7f1f3c004810 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c00bb90 con 0x7f1f3c0039f0 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1f3c00bd90 con 0x7f1f54074390 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f51d9b640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c004810 0x7f1f3c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5259c640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f3c0039f0 0x7f1f3c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f52d9d640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f54074390 0x7f1f3c0052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f51d9b640 1 -- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f54074390 msgr2=0x7f1f3c0052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:38.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f51d9b640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f54074390 0x7f1f3c0052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f51d9b640 1 -- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f3c0039f0 msgr2=0x7f1f3c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:38.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f51d9b640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f3c0039f0 0x7f1f3c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f51d9b640 1 -- 172.21.15.7:0/3640009554 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f3c003530 con 0x7f1f3c004810 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f51d9b640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c004810 0x7f1f3c004c90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1f44001780 tx=0x7f1f4401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb092a3341426106 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f440013a0 con 0x7f1f3c004810 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f440052f0 con 0x7f1f3c004810 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.655+0000 7f1f5159a640 1 -- 172.21.15.7:0/3640009554 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44003840 con 0x7f1f3c004810 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- 172.21.15.7:0/3640009554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c004810 msgr2=0x7f1f3c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 --2- 172.21.15.7:0/3640009554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c004810 0x7f1f3c004c90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1f44001780 tx=0x7f1f4401c420 comp rx=0 tx=0).stop 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f52d9d640 1 -- 172.21.15.7:0/3640009554 reap_dead start 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- 172.21.15.7:0/3640009554 shutdown_connections 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- 172.21.15.7:0/3640009554 >> 172.21.15.7:0/3640009554 conn(0x7f1f540b5bf0 msgr2=0x7f1f540b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- 172.21.15.7:0/3640009554 shutdown_connections 2024-03-19T01:23:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- 172.21.15.7:0/3640009554 wait complete. 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 Processor -- start 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- start start 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f3c004810 0x7f1f5414a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f540b1ca0 0x7f1f540b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f5413c090 0x7f1f54131370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1f54153e10 con 0x7f1f540b1ca0 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1f54153c90 con 0x7f1f5413c090 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1f54153f90 con 0x7f1f3c004810 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f51d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f540b1ca0 0x7f1f540b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f51d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f540b1ca0 0x7f1f540b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40344/0 (socket says 172.21.15.7:40344) 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f51d9b640 1 -- 172.21.15.7:0/3337043442 learned_addr learned my addr 172.21.15.7:0/3337043442 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f5259c640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f3c004810 0x7f1f5414a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f52d9d640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f5413c090 0x7f1f54131370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f51d9b640 1 -- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f3c004810 msgr2=0x7f1f5414a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f51d9b640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f3c004810 0x7f1f5414a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f51d9b640 1 -- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f5413c090 msgr2=0x7f1f54131370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f51d9b640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f5413c090 0x7f1f54131370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f51d9b640 1 -- 172.21.15.7:0/3337043442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5414aff0 con 0x7f1f540b1ca0 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f5259c640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f3c004810 0x7f1f5414a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f52d9d640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f5413c090 0x7f1f54131370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f51d9b640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f540b1ca0 0x7f1f540b2f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1f44005040 tx=0x7f1f3c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f42ffd640 1 -- 172.21.15.7:0/3337043442 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44007a40 con 0x7f1f540b1ca0 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f42ffd640 1 -- 172.21.15.7:0/3337043442 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f440bb070 con 0x7f1f540b1ca0 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f5413b480 con 0x7f1f540b1ca0 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f42ffd640 1 -- 172.21.15.7:0/3337043442 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44000ee0 con 0x7f1f540b1ca0 2024-03-19T01:23:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f5413bc00 con 0x7f1f540b1ca0 2024-03-19T01:23:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.659+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f54075460 con 0x7f1f540b1ca0 2024-03-19T01:23:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.663+0000 7f1f42ffd640 1 -- 172.21.15.7:0/3337043442 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1f440b4040 con 0x7f1f540b1ca0 2024-03-19T01:23:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.663+0000 7f1f42ffd640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f34067800 0x7f1f34069cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.663+0000 7f1f42ffd640 1 -- 172.21.15.7:0/3337043442 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1f4411f760 con 0x7f1f540b1ca0 2024-03-19T01:23:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.663+0000 7f1f5259c640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f34067800 0x7f1f34069cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.663+0000 7f1f5259c640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f34067800 0x7f1f34069cc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009a20 tx=0x7f1f48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.667+0000 7f1f42ffd640 1 -- 172.21.15.7:0/3337043442 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1f44016130 con 0x7f1f540b1ca0 2024-03-19T01:23:38.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f54145070 con 0x7f1f540b1ca0 2024-03-19T01:23:38.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f42ffd640 1 -- 172.21.15.7:0/3337043442 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7f1f440f0470 con 0x7f1f540b1ca0 2024-03-19T01:23:38.833 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:38.833 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f34067800 msgr2=0x7f1f34069cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f34067800 0x7f1f34069cc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009a20 tx=0x7f1f48008040 comp rx=0 tx=0).stop 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f540b1ca0 msgr2=0x7f1f540b2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f540b1ca0 0x7f1f540b2f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1f44005040 tx=0x7f1f3c005840 comp rx=0 tx=0).stop 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 shutdown_connections 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f34067800 0x7f1f34069cc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f5413c090 0x7f1f54131370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f540b1ca0 0x7f1f540b2f80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 --2- 172.21.15.7:0/3337043442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f3c004810 0x7f1f5414a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 >> 172.21.15.7:0/3337043442 conn(0x7f1f540b5bf0 msgr2=0x7f1f540b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 shutdown_connections 2024-03-19T01:23:38.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.827+0000 7f1f58bef640 1 -- 172.21.15.7:0/3337043442 wait complete. 2024-03-19T01:23:38.850 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:38.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:38.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.975+0000 7f1c6e398640 1 Processor -- start 2024-03-19T01:23:38.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6e398640 1 -- start start 2024-03-19T01:23:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6e398640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 0x7f1c680af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6e398640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1c6813dff0 0x7f1c6813e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6e398640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c680b9f90 con 0x7f1c680b9bb0 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6e398640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f1c680b5e30 con 0x7f1c6813dff0 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6e398640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1c680b62c0 con 0x7f1c680afb30 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c67fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1c680b9bb0 0x7f1c68147670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43312/0 (socket says 172.21.15.7:43312) 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c67fff640 1 -- 172.21.15.7:0/2848868828 learned_addr learned my addr 172.21.15.7:0/2848868828 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6c935640 1 --2- 172.21.15.7:0/2848868828 >> v2:172.21.15.7:3300/0 conn(0x7f1c6813dff0 0x7f1c6813e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c677fe640 1 --2- 172.21.15.7:0/2848868828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 0x7f1c680af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c66ffd640 1 -- 172.21.15.7:0/2848868828 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923985694 0 0) 0x7f1c680b9f90 con 0x7f1c680b9bb0 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c66ffd640 1 -- 172.21.15.7:0/2848868828 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c54003530 con 0x7f1c680b9bb0 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6c935640 1 -- 172.21.15.7:0/2848868828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 msgr2=0x7f1c680af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6c935640 1 --2- 172.21.15.7:0/2848868828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 0x7f1c680af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6c935640 1 -- 172.21.15.7:0/2848868828 >> v1:172.21.15.7:6789/0 conn(0x7f1c680b9bb0 legacy=0x7f1c68147670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.979+0000 7f1c6c935640 1 -- 172.21.15.7:0/2848868828 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c680b5ac0 con 0x7f1c6813dff0 2024-03-19T01:23:38.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 --2- 172.21.15.7:0/2848868828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 0x7f1c6813e450 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1c580097c0 tx=0x7f1c5801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cd10fff0fa9620 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c66ffd640 1 -- 172.21.15.7:0/2848868828 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c580b0070 con 0x7f1c6813dff0 2024-03-19T01:23:38.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c66ffd640 1 -- 172.21.15.7:0/2848868828 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c580ab600 con 0x7f1c6813dff0 2024-03-19T01:23:38.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c66ffd640 1 -- 172.21.15.7:0/2848868828 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c580ab9d0 con 0x7f1c6813dff0 2024-03-19T01:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- 172.21.15.7:0/2848868828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 msgr2=0x7f1c6813e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 --2- 172.21.15.7:0/2848868828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 0x7f1c6813e450 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1c580097c0 tx=0x7f1c5801c380 comp rx=0 tx=0).stop 2024-03-19T01:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- 172.21.15.7:0/2848868828 shutdown_connections 2024-03-19T01:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 --2- 172.21.15.7:0/2848868828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 0x7f1c6813e450 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 --2- 172.21.15.7:0/2848868828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 0x7f1c680af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- 172.21.15.7:0/2848868828 >> 172.21.15.7:0/2848868828 conn(0x7f1c6806e1b0 msgr2=0x7f1c6806e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- 172.21.15.7:0/2848868828 shutdown_connections 2024-03-19T01:23:38.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- 172.21.15.7:0/2848868828 wait complete. 2024-03-19T01:23:38.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 Processor -- start 2024-03-19T01:23:38.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- start start 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 0x7f1c6809f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c680b9bb0 0x7f1c681e2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 0x7f1c68136c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1c68151550 con 0x7f1c6813dff0 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1c681513d0 con 0x7f1c680b9bb0 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1c681516d0 con 0x7f1c680afb30 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 0x7f1c68136c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 0x7f1c68136c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40354/0 (socket says 172.21.15.7:40354) 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 -- 172.21.15.7:0/1251213193 learned_addr learned my addr 172.21.15.7:0/1251213193 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c67fff640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 0x7f1c6809f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 -- 172.21.15.7:0/1251213193 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 msgr2=0x7f1c6809f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c677fe640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c680b9bb0 0x7f1c681e2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 0x7f1c6809f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 -- 172.21.15.7:0/1251213193 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c680b9bb0 msgr2=0x7f1c681e2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c680b9bb0 0x7f1c681e2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 -- 172.21.15.7:0/1251213193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c681431a0 con 0x7f1c6813dff0 2024-03-19T01:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c67fff640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 0x7f1c6809f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6c935640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 0x7f1c68136c50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1c580ae040 tx=0x7f1c58019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c64ff9640 1 -- 172.21.15.7:0/1251213193 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c580b0040 con 0x7f1c6813dff0 2024-03-19T01:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c64ff9640 1 -- 172.21.15.7:0/1251213193 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c580b9070 con 0x7f1c6813dff0 2024-03-19T01:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c68072b10 con 0x7f1c6813dff0 2024-03-19T01:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c64ff9640 1 -- 172.21.15.7:0/1251213193 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c580b4650 con 0x7f1c6813dff0 2024-03-19T01:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.983+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c68071b90 con 0x7f1c6813dff0 2024-03-19T01:23:38.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.987+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c68075570 con 0x7f1c6813dff0 2024-03-19T01:23:38.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.987+0000 7f1c64ff9640 1 -- 172.21.15.7:0/1251213193 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1c580b47f0 con 0x7f1c6813dff0 2024-03-19T01:23:38.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.987+0000 7f1c64ff9640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1c48067660 0x7f1c48069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:38.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.987+0000 7f1c64ff9640 1 -- 172.21.15.7:0/1251213193 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1c5811ffe0 con 0x7f1c6813dff0 2024-03-19T01:23:38.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.987+0000 7f1c67fff640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1c48067660 0x7f1c48069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:38.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.987+0000 7f1c67fff640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1c48067660 0x7f1c48069b20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1c60005c10 tx=0x7f1c60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:38.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:38.991+0000 7f1c64ff9640 1 -- 172.21.15.7:0/1251213193 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1c58016090 con 0x7f1c6813dff0 2024-03-19T01:23:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.143+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1c68141d50 con 0x7f1c6813dff0 2024-03-19T01:23:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.143+0000 7f1c64ff9640 1 -- 172.21.15.7:0/1251213193 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f1c580f0d10 con 0x7f1c6813dff0 2024-03-19T01:23:39.150 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1c48067660 msgr2=0x7f1c48069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1c48067660 0x7f1c48069b20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1c60005c10 tx=0x7f1c60009040 comp rx=0 tx=0).stop 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 msgr2=0x7f1c68136c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 0x7f1c68136c50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1c580ae040 tx=0x7f1c58019120 comp rx=0 tx=0).stop 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 shutdown_connections 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1c48067660 0x7f1c48069b20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c6813dff0 0x7f1c68136c50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c680b9bb0 0x7f1c681e2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 --2- 172.21.15.7:0/1251213193 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c680afb30 0x7f1c6809f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 >> 172.21.15.7:0/1251213193 conn(0x7f1c6806e1b0 msgr2=0x7f1c680b81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 shutdown_connections 2024-03-19T01:23:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.147+0000 7f1c6e398640 1 -- 172.21.15.7:0/1251213193 wait complete. 2024-03-19T01:23:39.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:39.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77b5d2f640 1 Processor -- start 2024-03-19T01:23:39.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77b5d2f640 1 -- start start 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77b5d2f640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f77b00b1e50 0x7f77b0144220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77b5d2f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b01484b0 con 0x7f77b00af7d0 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77b5d2f640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b00b5990 con 0x7f77b00b15f0 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77b5d2f640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f77b00b5d00 con 0x7f77b00b1e50 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77aeffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f77b00af7d0 0x7f77b0147690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43328/0 (socket says 172.21.15.7:43328) 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77aeffd640 1 -- 172.21.15.7:0/4259623068 learned_addr learned my addr 172.21.15.7:0/4259623068 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77af7fe640 1 --2- 172.21.15.7:0/4259623068 >> v2:172.21.15.167:3300/0 conn(0x7f77b00b1e50 0x7f77b0144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4165239417 0 0) 0x7f77b01484b0 con 0x7f77b00af7d0 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7798003530 con 0x7f77b00af7d0 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3160832676 0 0) 0x7f7798003530 con 0x7f77b00af7d0 2024-03-19T01:23:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 >> v2:172.21.15.167:3300/0 conn(0x7f77b00b1e50 msgr2=0x7f77b0144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 --2- 172.21.15.7:0/4259623068 >> v2:172.21.15.167:3300/0 conn(0x7f77b00b1e50 0x7f77b0144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 >> v1:172.21.15.167:6789/0 conn(0x7f77b00b15f0 legacy=0x7f77b0147da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b00b2230 con 0x7f77b00af7d0 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f77a4001680 con 0x7f77b00af7d0 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 >> v1:172.21.15.7:6789/0 conn(0x7f77b00af7d0 legacy=0x7f77b0147690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77980039c0 0x7f7798003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7798004810 0x7f7798004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77980051d0 0x7f779800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f779800c140 con 0x7f7798004810 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f779800c2f0 con 0x7f77980039c0 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f779800e700 con 0x7f77980051d0 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77aeffd640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7798004810 0x7f7798004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77af7fe640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77980039c0 0x7f7798003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77affff640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77980051d0 0x7f779800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77aeffd640 1 -- 172.21.15.7:0/4259623068 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77980051d0 msgr2=0x7f779800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:39.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77aeffd640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77980051d0 0x7f779800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77aeffd640 1 -- 172.21.15.7:0/4259623068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77980039c0 msgr2=0x7f7798003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:39.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77aeffd640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77980039c0 0x7f7798003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77aeffd640 1 -- 172.21.15.7:0/4259623068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7798003530 con 0x7f7798004810 2024-03-19T01:23:39.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.283+0000 7f77affff640 1 -- 172.21.15.7:0/4259623068 reap_dead start 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77aeffd640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7798004810 0x7f7798004c90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f77b00b6220 tx=0x7f77a401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49765ba4cd4efe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4007d40 con 0x7f7798004810 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77a40a9070 con 0x7f7798004810 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77ae7fc640 1 -- 172.21.15.7:0/4259623068 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a40a5da0 con 0x7f7798004810 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- 172.21.15.7:0/4259623068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7798004810 msgr2=0x7f7798004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7798004810 0x7f7798004c90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f77b00b6220 tx=0x7f77a401cc60 comp rx=0 tx=0).stop 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- 172.21.15.7:0/4259623068 shutdown_connections 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 --2- 172.21.15.7:0/4259623068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7798004810 0x7f7798004c90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- 172.21.15.7:0/4259623068 >> 172.21.15.7:0/4259623068 conn(0x7f77b006c3b0 msgr2=0x7f77b006c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:39.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- 172.21.15.7:0/4259623068 shutdown_connections 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- 172.21.15.7:0/4259623068 wait complete. 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 Processor -- start 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- start start 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7798004810 0x7f77b00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77b01db6b0 0x7f77b01dbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b01deca0 0x7f77b01df120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f77b0148230 con 0x7f77b01deca0 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f77b01480b0 con 0x7f7798004810 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f77b013da00 con 0x7f77b01db6b0 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77affff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b01deca0 0x7f77b01df120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77affff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b01deca0 0x7f77b01df120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40370/0 (socket says 172.21.15.7:40370) 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77affff640 1 -- 172.21.15.7:0/888794923 learned_addr learned my addr 172.21.15.7:0/888794923 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77af7fe640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7798004810 0x7f77b00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77aeffd640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77b01db6b0 0x7f77b01dbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77affff640 1 -- 172.21.15.7:0/888794923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77b01db6b0 msgr2=0x7f77b01dbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77affff640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77b01db6b0 0x7f77b01dbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77affff640 1 -- 172.21.15.7:0/888794923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7798004810 msgr2=0x7f77b00b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77affff640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7798004810 0x7f77b00b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77affff640 1 -- 172.21.15.7:0/888794923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b00b92b0 con 0x7f77b01deca0 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77aeffd640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77b01db6b0 0x7f77b01dbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77affff640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b01deca0 0x7f77b01df120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f77b00b1a00 tx=0x7f77a800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f7797fff640 1 -- 172.21.15.7:0/888794923 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a8018070 con 0x7f77b01deca0 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f7797fff640 1 -- 172.21.15.7:0/888794923 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77a8003b80 con 0x7f77b01deca0 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b00b8290 con 0x7f77b01deca0 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f7797fff640 1 -- 172.21.15.7:0/888794923 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a8005070 con 0x7f77b01deca0 2024-03-19T01:23:39.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.287+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b00b7d70 con 0x7f77b01deca0 2024-03-19T01:23:39.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.291+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b00b7f70 con 0x7f77b01deca0 2024-03-19T01:23:39.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.291+0000 7f7797fff640 1 -- 172.21.15.7:0/888794923 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f77a801b070 con 0x7f77b01deca0 2024-03-19T01:23:39.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.291+0000 7f7797fff640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f779c067690 0x7f779c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.291+0000 7f7797fff640 1 -- 172.21.15.7:0/888794923 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f77a8084e70 con 0x7f77b01deca0 2024-03-19T01:23:39.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.291+0000 7f77af7fe640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f779c067690 0x7f779c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.291+0000 7f77af7fe640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f779c067690 0x7f779c069b50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009ab0 tx=0x7f77a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:39.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.295+0000 7f7797fff640 1 -- 172.21.15.7:0/888794923 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f77b00b7f70 con 0x7f77b01deca0 2024-03-19T01:23:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.451+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f77b0075570 con 0x7f77b01deca0 2024-03-19T01:23:39.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.451+0000 7f7797fff640 1 -- 172.21.15.7:0/888794923 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f77a8055c40 con 0x7f77b01deca0 2024-03-19T01:23:39.458 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f779c067690 msgr2=0x7f779c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f779c067690 0x7f779c069b50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009ab0 tx=0x7f77a0008040 comp rx=0 tx=0).stop 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b01deca0 msgr2=0x7f77b01df120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b01deca0 0x7f77b01df120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f77b00b1a00 tx=0x7f77a800dc00 comp rx=0 tx=0).stop 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 shutdown_connections 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f779c067690 0x7f779c069b50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77b01db6b0 0x7f77b01dbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b01deca0 0x7f77b01df120 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 --2- 172.21.15.7:0/888794923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7798004810 0x7f77b00b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 >> 172.21.15.7:0/888794923 conn(0x7f77b006c3b0 msgr2=0x7f77b013d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 shutdown_connections 2024-03-19T01:23:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.455+0000 7f77b5d2f640 1 -- 172.21.15.7:0/888794923 wait complete. 2024-03-19T01:23:39.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:39.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a18c69640 1 Processor -- start 2024-03-19T01:23:39.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a18c69640 1 -- start start 2024-03-19T01:23:39.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a18c69640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f7a140b4ce0 0x7f7a14147880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a18c69640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a140b6280 con 0x7f7a1406b520 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a18c69640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a140af9d0 con 0x7f7a14147dc0 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a18c69640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f7a1406b900 con 0x7f7a140b4ce0 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1259c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7a1406b520 0x7f7a140af2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43332/0 (socket says 172.21.15.7:43332) 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1259c640 1 -- 172.21.15.7:0/2486439744 learned_addr learned my addr 172.21.15.7:0/2486439744 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a11d9b640 1 --2- 172.21.15.7:0/2486439744 >> v2:172.21.15.167:3300/0 conn(0x7f7a140b4ce0 0x7f7a14147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2110912972 0 0) 0x7f7a0c001120 con 0x7f7a1406b520 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79fc003530 con 0x7f7a1406b520 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1071707793 0 0) 0x7f79fc003530 con 0x7f7a1406b520 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 >> v2:172.21.15.167:3300/0 conn(0x7f7a140b4ce0 msgr2=0x7f7a14147880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 --2- 172.21.15.7:0/2486439744 >> v2:172.21.15.167:3300/0 conn(0x7f7a140b4ce0 0x7f7a14147880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 >> v1:172.21.15.167:6789/0 conn(0x7f7a14147dc0 legacy=0x7f7a141520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a140b54e0 con 0x7f7a1406b520 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f7a0c001850 con 0x7f7a1406b520 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 >> v1:172.21.15.7:6789/0 conn(0x7f7a1406b520 legacy=0x7f7a140af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79fc0039c0 0x7f79fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79fc004810 0x7f79fc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79fc0052f0 0x7f79fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1406b900 con 0x7f79fc0039c0 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc00c1c0 con 0x7f79fc0052f0 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f79fc00c3c0 con 0x7f79fc004810 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1259c640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79fc0039c0 0x7f79fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a12d9d640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79fc0052f0 0x7f79fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1259c640 1 -- 172.21.15.7:0/2486439744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79fc004810 msgr2=0x7f79fc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1259c640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79fc004810 0x7f79fc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1259c640 1 -- 172.21.15.7:0/2486439744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79fc0052f0 msgr2=0x7f79fc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1259c640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79fc0052f0 0x7f79fc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a1259c640 1 -- 172.21.15.7:0/2486439744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79fc003530 con 0x7f79fc0039c0 2024-03-19T01:23:39.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a12d9d640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79fc0052f0 0x7f79fc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:39.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.595+0000 7f7a12d9d640 1 -- 172.21.15.7:0/2486439744 reap_dead start 2024-03-19T01:23:39.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a1259c640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79fc0039c0 0x7f79fc003de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c490 tx=0x7f7a0c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288b242febbdb2c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:39.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c00c0e0 con 0x7f79fc0039c0 2024-03-19T01:23:39.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0c0a9020 con 0x7f79fc0039c0 2024-03-19T01:23:39.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a1159a640 1 -- 172.21.15.7:0/2486439744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c005b00 con 0x7f79fc0039c0 2024-03-19T01:23:39.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 -- 172.21.15.7:0/2486439744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79fc0039c0 msgr2=0x7f79fc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79fc0039c0 0x7f79fc003de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c490 tx=0x7f7a0c01cc40 comp rx=0 tx=0).stop 2024-03-19T01:23:39.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 -- 172.21.15.7:0/2486439744 shutdown_connections 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 --2- 172.21.15.7:0/2486439744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79fc0039c0 0x7f79fc003de0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 -- 172.21.15.7:0/2486439744 >> 172.21.15.7:0/2486439744 conn(0x7f7a1406e1b0 msgr2=0x7f7a1406e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 -- 172.21.15.7:0/2486439744 shutdown_connections 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 -- 172.21.15.7:0/2486439744 wait complete. 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 Processor -- start 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 -- start start 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79fc0039c0 0x7f7a141e3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a140b0f90 0x7f7a14071810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a1413e7f0 0x7f7a1413ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1409fc30 con 0x7f7a140b0f90 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7a140b5b00 con 0x7f7a1413e7f0 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a18c69640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7a140b1ef0 con 0x7f79fc0039c0 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a11d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a140b0f90 0x7f7a14071810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a11d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a140b0f90 0x7f7a14071810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40380/0 (socket says 172.21.15.7:40380) 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a11d9b640 1 -- 172.21.15.7:0/2899154471 learned_addr learned my addr 172.21.15.7:0/2899154471 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a1259c640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79fc0039c0 0x7f7a141e3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a12d9d640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a1413e7f0 0x7f7a1413ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a11d9b640 1 -- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79fc0039c0 msgr2=0x7f7a141e3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a11d9b640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79fc0039c0 0x7f7a141e3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a11d9b640 1 -- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a1413e7f0 msgr2=0x7f7a1413ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a11d9b640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a1413e7f0 0x7f7a1413ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.599+0000 7f7a11d9b640 1 -- 172.21.15.7:0/2899154471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a140b13f0 con 0x7f7a140b0f90 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a12d9d640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a1413e7f0 0x7f7a1413ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a1259c640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79fc0039c0 0x7f7a141e3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a11d9b640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a140b0f90 0x7f7a14071810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800e840 tx=0x7f7a0800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a02ffd640 1 -- 172.21.15.7:0/2899154471 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a08017070 con 0x7f7a140b0f90 2024-03-19T01:23:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a02ffd640 1 -- 172.21.15.7:0/2899154471 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0800c940 con 0x7f7a140b0f90 2024-03-19T01:23:39.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a141419c0 con 0x7f7a140b0f90 2024-03-19T01:23:39.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a02ffd640 1 -- 172.21.15.7:0/2899154471 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0800cd30 con 0x7f7a140b0f90 2024-03-19T01:23:39.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a14141fb0 con 0x7f7a140b0f90 2024-03-19T01:23:39.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a14075570 con 0x7f7a140b0f90 2024-03-19T01:23:39.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a02ffd640 1 -- 172.21.15.7:0/2899154471 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7a0801a070 con 0x7f7a140b0f90 2024-03-19T01:23:39.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a02ffd640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f79f4067690 0x7f79f4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a02ffd640 1 -- 172.21.15.7:0/2899154471 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7a080856a0 con 0x7f7a140b0f90 2024-03-19T01:23:39.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a1259c640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f79f4067690 0x7f79f4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.603+0000 7f7a1259c640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f79f4067690 0x7f79f4069b50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0aa000 tx=0x7f7a140b05f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:39.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.607+0000 7f7a02ffd640 1 -- 172.21.15.7:0/2899154471 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7a080563b0 con 0x7f7a140b0f90 2024-03-19T01:23:39.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.759+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7a140b2160 con 0x7f7a140b0f90 2024-03-19T01:23:39.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.759+0000 7f7a02ffd640 1 -- 172.21.15.7:0/2899154471 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f7a08055cb0 con 0x7f7a140b0f90 2024-03-19T01:23:39.767 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f79f4067690 msgr2=0x7f79f4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f79f4067690 0x7f79f4069b50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0aa000 tx=0x7f7a140b05f0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a140b0f90 msgr2=0x7f7a14071810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a140b0f90 0x7f7a14071810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800e840 tx=0x7f7a0800ed00 comp rx=0 tx=0).stop 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 shutdown_connections 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f79f4067690 0x7f79f4069b50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a1413e7f0 0x7f7a1413ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a140b0f90 0x7f7a14071810 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 --2- 172.21.15.7:0/2899154471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79fc0039c0 0x7f7a141e3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 >> 172.21.15.7:0/2899154471 conn(0x7f7a1406e1b0 msgr2=0x7f7a1414b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:39.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 shutdown_connections 2024-03-19T01:23:39.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.763+0000 7f7a18c69640 1 -- 172.21.15.7:0/2899154471 wait complete. 2024-03-19T01:23:39.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:39.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f2445eb0640 1 Processor -- start 2024-03-19T01:23:39.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f2445eb0640 1 -- start start 2024-03-19T01:23:39.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f2445eb0640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2440074390 0x7f244013e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f2445eb0640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2440144ce0 0x7f2440147e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f2445eb0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2440074770 con 0x7f2440144880 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f2445eb0640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f24400b05f0 con 0x7f2440144ce0 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f2445eb0640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f24400702a0 con 0x7f2440074390 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2440144ce0 0x7f2440147e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243effd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2440144880 0x7f2440147740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43342/0 (socket says 172.21.15.7:43342) 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243effd640 1 -- 172.21.15.7:0/1252683776 learned_addr learned my addr 172.21.15.7:0/1252683776 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2440144ce0 0x7f2440147e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40382/0 (socket says 172.21.15.7:40382) 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243f7fe640 1 --2- 172.21.15.7:0/1252683776 >> v2:172.21.15.167:3300/0 conn(0x7f2440074390 0x7f244013e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243ffff640 1 -- 172.21.15.7:0/1252683776 >> v2:172.21.15.167:3300/0 conn(0x7f2440074390 msgr2=0x7f244013e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243ffff640 1 --2- 172.21.15.7:0/1252683776 >> v2:172.21.15.167:3300/0 conn(0x7f2440074390 0x7f244013e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243ffff640 1 -- 172.21.15.7:0/1252683776 >> v1:172.21.15.7:6789/0 conn(0x7f2440144880 legacy=0x7f2440147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243e7fc640 1 -- 172.21.15.7:0/1252683776 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 606231564 0 0) 0x7f2440074770 con 0x7f2440144880 2024-03-19T01:23:39.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.907+0000 7f243ffff640 1 -- 172.21.15.7:0/1252683776 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24400b2d30 con 0x7f2440144ce0 2024-03-19T01:23:39.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 --2- 172.21.15.7:0/1252683776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 0x7f2440147e50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f24400b86c0 tx=0x7f243801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c811962b54f0c723 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:39.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243e7fc640 1 -- 172.21.15.7:0/1252683776 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24380b2070 con 0x7f2440144ce0 2024-03-19T01:23:39.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243e7fc640 1 -- 172.21.15.7:0/1252683776 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24380a5760 con 0x7f2440144ce0 2024-03-19T01:23:39.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243e7fc640 1 -- 172.21.15.7:0/1252683776 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24380a5b30 con 0x7f2440144ce0 2024-03-19T01:23:39.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- 172.21.15.7:0/1252683776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 msgr2=0x7f2440147e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 --2- 172.21.15.7:0/1252683776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 0x7f2440147e50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f24400b86c0 tx=0x7f243801c8a0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- 172.21.15.7:0/1252683776 shutdown_connections 2024-03-19T01:23:39.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 --2- 172.21.15.7:0/1252683776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 0x7f2440147e50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 --2- 172.21.15.7:0/1252683776 >> v2:172.21.15.167:3300/0 conn(0x7f2440074390 0x7f244013e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- 172.21.15.7:0/1252683776 >> 172.21.15.7:0/1252683776 conn(0x7f24400b9170 msgr2=0x7f24400b9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:39.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- 172.21.15.7:0/1252683776 shutdown_connections 2024-03-19T01:23:39.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- 172.21.15.7:0/1252683776 wait complete. 2024-03-19T01:23:39.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 Processor -- start 2024-03-19T01:23:39.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- start start 2024-03-19T01:23:39.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2440074390 0x7f244013e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2440144880 0x7f2440138830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 0x7f2440138d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2440148390 con 0x7f2440144ce0 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2440145130 con 0x7f2440074390 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2440148510 con 0x7f2440144880 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 0x7f2440138d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 0x7f2440138d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40396/0 (socket says 172.21.15.7:40396) 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 -- 172.21.15.7:0/1324820491 learned_addr learned my addr 172.21.15.7:0/1324820491 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 -- 172.21.15.7:0/1324820491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2440144880 msgr2=0x7f2440138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2440144880 0x7f2440138830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 -- 172.21.15.7:0/1324820491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2440074390 msgr2=0x7f244013e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2440074390 0x7f244013e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 -- 172.21.15.7:0/1324820491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244013a180 con 0x7f2440144ce0 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f243ffff640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 0x7f2440138d70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f243801a4d0 tx=0x7f2438019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2427fff640 1 -- 172.21.15.7:0/1324820491 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24380b2050 con 0x7f2440144ce0 2024-03-19T01:23:39.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2427fff640 1 -- 172.21.15.7:0/1324820491 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24380be070 con 0x7f2440144ce0 2024-03-19T01:23:39.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244013e690 con 0x7f2440144ce0 2024-03-19T01:23:39.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2427fff640 1 -- 172.21.15.7:0/1324820491 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24380ad870 con 0x7f2440144ce0 2024-03-19T01:23:39.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.911+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244013b6f0 con 0x7f2440144ce0 2024-03-19T01:23:39.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.915+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2440074a70 con 0x7f2440144ce0 2024-03-19T01:23:39.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.915+0000 7f2427fff640 1 -- 172.21.15.7:0/1324820491 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f24380ada10 con 0x7f2440144ce0 2024-03-19T01:23:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.915+0000 7f2427fff640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f242c067660 0x7f242c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.915+0000 7f2427fff640 1 -- 172.21.15.7:0/1324820491 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2438120b00 con 0x7f2440144ce0 2024-03-19T01:23:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.915+0000 7f243f7fe640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f242c067660 0x7f242c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:39.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.915+0000 7f243f7fe640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f242c067660 0x7f242c069b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2440142090 tx=0x7f2430008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:39.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:39.919+0000 7f2427fff640 1 -- 172.21.15.7:0/1324820491 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2438016520 con 0x7f2440144ce0 2024-03-19T01:23:40.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.107+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2440075460 con 0x7f2440144ce0 2024-03-19T01:23:40.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.107+0000 7f2427fff640 1 -- 172.21.15.7:0/1324820491 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3524 (secure 0 0 0) 0x7f24380add00 con 0x7f2440144ce0 2024-03-19T01:23:40.113 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:40.114 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4208,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/656985991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":656985991},{"type":"v1","addr":"172.21.15.167:6833","nonce":656985991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:23:26.868603+0000","modified":"2024-03-19T01:23:29.918429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-03-19T01:23:40.114 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f242c067660 msgr2=0x7f242c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f242c067660 0x7f242c069b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2440142090 tx=0x7f2430008040 comp rx=0 tx=0).stop 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 msgr2=0x7f2440138d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 0x7f2440138d70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f243801a4d0 tx=0x7f2438019640 comp rx=0 tx=0).stop 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 shutdown_connections 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f242c067660 0x7f242c069b20 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2440142090 tx=0x7f2430008040 comp rx=0 tx=0).stop 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2440144880 0x7f2440138830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2440144ce0 0x7f2440138d70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 --2- 172.21.15.7:0/1324820491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2440074390 0x7f244013e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 >> 172.21.15.7:0/1324820491 conn(0x7f24400b9170 msgr2=0x7f24400b7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 shutdown_connections 2024-03-19T01:23:40.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.111+0000 7f2445eb0640 1 -- 172.21.15.7:0/1324820491 wait complete. 2024-03-19T01:23:40.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:40.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.239+0000 7fea39fb5640 1 Processor -- start 2024-03-19T01:23:40.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.239+0000 7fea39fb5640 1 -- start start 2024-03-19T01:23:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea39fb5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fea34071b60 0x7fea340b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea39fb5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea341454f0 con 0x7fea34145cf0 2024-03-19T01:23:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea39fb5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea341451e0 con 0x7fea34071780 2024-03-19T01:23:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea39fb5640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fea34145860 con 0x7fea34071b60 2024-03-19T01:23:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea32ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fea34071b60 0x7fea340b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea33fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fea34145cf0 0x7fea3413ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43354/0 (socket says 172.21.15.7:43354) 2024-03-19T01:23:40.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea33fff640 1 -- 172.21.15.7:0/2640487178 learned_addr learned my addr 172.21.15.7:0/2640487178 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:40.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea32ffd640 1 -- 172.21.15.7:0/2640487178 >> v1:172.21.15.167:6789/0 conn(0x7fea34071780 legacy=0x7fea34072da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:40.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea32ffd640 1 -- 172.21.15.7:0/2640487178 >> v1:172.21.15.7:6789/0 conn(0x7fea34145cf0 legacy=0x7fea3413ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea32ffd640 1 -- 172.21.15.7:0/2640487178 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3413f660 con 0x7fea34071b60 2024-03-19T01:23:40.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.243+0000 7fea327fc640 1 -- 172.21.15.7:0/2640487178 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2638964650 0 0) 0x7fea341454f0 con 0x7fea34145cf0 2024-03-19T01:23:40.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea32ffd640 1 --2- 172.21.15.7:0/2640487178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071b60 0x7fea340b5460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fea240097c0 tx=0x7fea2401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8474b60c02e716e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:40.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea327fc640 1 -- 172.21.15.7:0/2640487178 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea240b0070 con 0x7fea34071b60 2024-03-19T01:23:40.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea327fc640 1 -- 172.21.15.7:0/2640487178 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea240ab600 con 0x7fea34071b60 2024-03-19T01:23:40.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea327fc640 1 -- 172.21.15.7:0/2640487178 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea240ab9d0 con 0x7fea34071b60 2024-03-19T01:23:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- 172.21.15.7:0/2640487178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071b60 msgr2=0x7fea340b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 --2- 172.21.15.7:0/2640487178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071b60 0x7fea340b5460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fea240097c0 tx=0x7fea2401c3d0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- 172.21.15.7:0/2640487178 shutdown_connections 2024-03-19T01:23:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 --2- 172.21.15.7:0/2640487178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071b60 0x7fea340b5460 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- 172.21.15.7:0/2640487178 >> 172.21.15.7:0/2640487178 conn(0x7fea3406bc10 msgr2=0x7fea340b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- 172.21.15.7:0/2640487178 shutdown_connections 2024-03-19T01:23:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- 172.21.15.7:0/2640487178 wait complete. 2024-03-19T01:23:40.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 Processor -- start 2024-03-19T01:23:40.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- start start 2024-03-19T01:23:40.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071780 0x7fea3406df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea34071b60 0x7fea341e2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fea34145cf0 0x7fea341e3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fea34146120 con 0x7fea34071780 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fea340b2650 con 0x7fea34145cf0 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fea341539b0 con 0x7fea34071b60 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea337fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071780 0x7fea3406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea337fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071780 0x7fea3406df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40426/0 (socket says 172.21.15.7:40426) 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea337fe640 1 -- 172.21.15.7:0/2804762984 learned_addr learned my addr 172.21.15.7:0/2804762984 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea32ffd640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea34071b60 0x7fea341e2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea337fe640 1 -- 172.21.15.7:0/2804762984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea34071b60 msgr2=0x7fea341e2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea337fe640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea34071b60 0x7fea341e2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea337fe640 1 -- 172.21.15.7:0/2804762984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fea34145cf0 msgr2=0x7fea341e3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea337fe640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fea34145cf0 0x7fea341e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea337fe640 1 -- 172.21.15.7:0/2804762984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea34135b90 con 0x7fea34071780 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea337fe640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071780 0x7fea3406df90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c480 tx=0x7fea2800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea1bfff640 1 -- 172.21.15.7:0/2804762984 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28012830 con 0x7fea34071780 2024-03-19T01:23:40.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea1bfff640 1 -- 172.21.15.7:0/2804762984 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea28017070 con 0x7fea34071780 2024-03-19T01:23:40.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea3413cfa0 con 0x7fea34071780 2024-03-19T01:23:40.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea1bfff640 1 -- 172.21.15.7:0/2804762984 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28012c30 con 0x7fea34071780 2024-03-19T01:23:40.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.247+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea3413c940 con 0x7fea34071780 2024-03-19T01:23:40.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.251+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea34075460 con 0x7fea34071780 2024-03-19T01:23:40.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.251+0000 7fea1bfff640 1 -- 172.21.15.7:0/2804762984 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fea2801c460 con 0x7fea34071780 2024-03-19T01:23:40.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.251+0000 7fea1bfff640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fea20067660 0x7fea20069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.251+0000 7fea1bfff640 1 -- 172.21.15.7:0/2804762984 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fea28085620 con 0x7fea34071780 2024-03-19T01:23:40.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.251+0000 7fea32ffd640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fea20067660 0x7fea20069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.251+0000 7fea32ffd640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fea20067660 0x7fea20069b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fea240ad000 tx=0x7fea24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:40.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.255+0000 7fea1bfff640 1 -- 172.21.15.7:0/2804762984 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fea28056330 con 0x7fea34071780 2024-03-19T01:23:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.407+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea3413e560 con 0x7fea34071780 2024-03-19T01:23:40.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.411+0000 7fea1bfff640 1 -- 172.21.15.7:0/2804762984 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7fea28055c30 con 0x7fea34071780 2024-03-19T01:23:40.417 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:40.417 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:40.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fea20067660 msgr2=0x7fea20069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fea20067660 0x7fea20069b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fea240ad000 tx=0x7fea24008040 comp rx=0 tx=0).stop 2024-03-19T01:23:40.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071780 msgr2=0x7fea3406df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071780 0x7fea3406df90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c480 tx=0x7fea2800c940 comp rx=0 tx=0).stop 2024-03-19T01:23:40.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 shutdown_connections 2024-03-19T01:23:40.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fea20067660 0x7fea20069b20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea34071b60 0x7fea341e2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fea34145cf0 0x7fea341e3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 --2- 172.21.15.7:0/2804762984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea34071780 0x7fea3406df90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 >> 172.21.15.7:0/2804762984 conn(0x7fea3406bc10 msgr2=0x7fea3413d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:40.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 shutdown_connections 2024-03-19T01:23:40.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.415+0000 7fea39fb5640 1 -- 172.21.15.7:0/2804762984 wait complete. 2024-03-19T01:23:40.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:40.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.535+0000 7fbffb680640 1 Processor -- start 2024-03-19T01:23:40.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbffb680640 1 -- start start 2024-03-19T01:23:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbffb680640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbff4074390 0x7fbff4148b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbffb680640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff40b4ec0 0x7fbff4149060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbffb680640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fbff413ef50 0x7fbff413f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbffb680640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fbff4070fd0 con 0x7fbff4074390 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbffb680640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fbff4070d60 con 0x7fbff413ef50 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbffb680640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbff4074770 con 0x7fbff40b4ec0 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff941c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbff4074390 0x7fbff4148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff941c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbff4074390 0x7fbff4148b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40428/0 (socket says 172.21.15.7:40428) 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff941c640 1 -- 172.21.15.7:0/3919565979 learned_addr learned my addr 172.21.15.7:0/3919565979 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff8c1b640 1 --2- 172.21.15.7:0/3919565979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff40b4ec0 0x7fbff4149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff9c1d640 1 --2- 172.21.15.7:0/3919565979 >> v2:172.21.15.167:3300/0 conn(0x7fbff413ef50 0x7fbff413f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff941c640 1 -- 172.21.15.7:0/3919565979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff40b4ec0 msgr2=0x7fbff4149060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff941c640 1 --2- 172.21.15.7:0/3919565979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff40b4ec0 0x7fbff4149060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff941c640 1 -- 172.21.15.7:0/3919565979 >> v2:172.21.15.167:3300/0 conn(0x7fbff413ef50 msgr2=0x7fbff413f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff941c640 1 --2- 172.21.15.7:0/3919565979 >> v2:172.21.15.167:3300/0 conn(0x7fbff413ef50 0x7fbff413f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff941c640 1 -- 172.21.15.7:0/3919565979 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff406fcc0 con 0x7fbff4074390 2024-03-19T01:23:40.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff9c1d640 1 --2- 172.21.15.7:0/3919565979 >> v2:172.21.15.167:3300/0 conn(0x7fbff413ef50 0x7fbff413f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:40.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.539+0000 7fbff941c640 1 --2- 172.21.15.7:0/3919565979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff4074390 0x7fbff4148b20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbff4073a80 tx=0x7fbff001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94b1fe3e10ca558 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:40.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbfebfff640 1 -- 172.21.15.7:0/3919565979 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff00b2070 con 0x7fbff4074390 2024-03-19T01:23:40.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbfebfff640 1 -- 172.21.15.7:0/3919565979 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff00a9070 con 0x7fbff4074390 2024-03-19T01:23:40.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbfebfff640 1 -- 172.21.15.7:0/3919565979 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff00ad880 con 0x7fbff4074390 2024-03-19T01:23:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 -- 172.21.15.7:0/3919565979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff4074390 msgr2=0x7fbff4148b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 --2- 172.21.15.7:0/3919565979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff4074390 0x7fbff4148b20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbff4073a80 tx=0x7fbff001c490 comp rx=0 tx=0).stop 2024-03-19T01:23:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 -- 172.21.15.7:0/3919565979 shutdown_connections 2024-03-19T01:23:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 --2- 172.21.15.7:0/3919565979 >> v2:172.21.15.167:3300/0 conn(0x7fbff413ef50 0x7fbff413f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 --2- 172.21.15.7:0/3919565979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff40b4ec0 0x7fbff4149060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 --2- 172.21.15.7:0/3919565979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff4074390 0x7fbff4148b20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 -- 172.21.15.7:0/3919565979 >> 172.21.15.7:0/3919565979 conn(0x7fbff40b5bf0 msgr2=0x7fbff40b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 -- 172.21.15.7:0/3919565979 shutdown_connections 2024-03-19T01:23:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 -- 172.21.15.7:0/3919565979 wait complete. 2024-03-19T01:23:40.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 Processor -- start 2024-03-19T01:23:40.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 -- start start 2024-03-19T01:23:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbff4074390 0x7fbff41e3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff40b4ec0 0x7fbff413e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff413ef50 0x7fbff40b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbff4145b40 con 0x7fbff40b4ec0 2024-03-19T01:23:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbff41459c0 con 0x7fbff4074390 2024-03-19T01:23:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.543+0000 7fbffb680640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbff4145cc0 con 0x7fbff413ef50 2024-03-19T01:23:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff8c1b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff40b4ec0 0x7fbff413e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff8c1b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff40b4ec0 0x7fbff413e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40434/0 (socket says 172.21.15.7:40434) 2024-03-19T01:23:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff8c1b640 1 -- 172.21.15.7:0/2443296495 learned_addr learned my addr 172.21.15.7:0/2443296495 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff941c640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbff4074390 0x7fbff41e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff8c1b640 1 -- 172.21.15.7:0/2443296495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff413ef50 msgr2=0x7fbff40b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff9c1d640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff413ef50 0x7fbff40b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff8c1b640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff413ef50 0x7fbff40b1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff8c1b640 1 -- 172.21.15.7:0/2443296495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbff4074390 msgr2=0x7fbff41e3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff8c1b640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbff4074390 0x7fbff41e3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff8c1b640 1 -- 172.21.15.7:0/2443296495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff4146e80 con 0x7fbff40b4ec0 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff941c640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbff4074390 0x7fbff41e3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff8c1b640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff40b4ec0 0x7fbff413e1a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400d830 tx=0x7fbfe400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbfe9ffb640 1 -- 172.21.15.7:0/2443296495 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400b6c0 con 0x7fbff40b4ec0 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbfe9ffb640 1 -- 172.21.15.7:0/2443296495 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe4019070 con 0x7fbff40b4ec0 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff40b1650 con 0x7fbff40b4ec0 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbfe9ffb640 1 -- 172.21.15.7:0/2443296495 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400ba60 con 0x7fbff40b4ec0 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff40b24d0 con 0x7fbff40b4ec0 2024-03-19T01:23:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff4075460 con 0x7fbff40b4ec0 2024-03-19T01:23:40.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbfe9ffb640 1 -- 172.21.15.7:0/2443296495 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbfe401e460 con 0x7fbff40b4ec0 2024-03-19T01:23:40.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbfe9ffb640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbfd8067690 0x7fbfd8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:40.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbfe9ffb640 1 -- 172.21.15.7:0/2443296495 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbfe4086930 con 0x7fbff40b4ec0 2024-03-19T01:23:40.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff941c640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbfd8067690 0x7fbfd8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:40.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.547+0000 7fbff941c640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbfd8067690 0x7fbfd8069b50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbff00af000 tx=0x7fbff0004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:40.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.555+0000 7fbfe9ffb640 1 -- 172.21.15.7:0/2443296495 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbfe4057720 con 0x7fbff40b4ec0 2024-03-19T01:23:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.747+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbff40b55f0 con 0x7fbff40b4ec0 2024-03-19T01:23:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.747+0000 7fbfe9ffb640 1 -- 172.21.15.7:0/2443296495 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3524 (secure 0 0 0) 0x7fbfe4057020 con 0x7fbff40b4ec0 2024-03-19T01:23:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:40.753 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4208,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/656985991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":656985991},{"type":"v1","addr":"172.21.15.167:6833","nonce":656985991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:23:26.868603+0000","modified":"2024-03-19T01:23:29.918429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-03-19T01:23:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-03-19T01:23:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbfd8067690 msgr2=0x7fbfd8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbfd8067690 0x7fbfd8069b50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbff00af000 tx=0x7fbff0004740 comp rx=0 tx=0).stop 2024-03-19T01:23:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff40b4ec0 msgr2=0x7fbff413e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff40b4ec0 0x7fbff413e1a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400d830 tx=0x7fbfe400dcf0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 shutdown_connections 2024-03-19T01:23:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbfd8067690 0x7fbfd8069b50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbff413ef50 0x7fbff40b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbff40b4ec0 0x7fbff413e1a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 --2- 172.21.15.7:0/2443296495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbff4074390 0x7fbff41e3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 >> 172.21.15.7:0/2443296495 conn(0x7fbff40b5bf0 msgr2=0x7fbff40b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 shutdown_connections 2024-03-19T01:23:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:40.751+0000 7fbffb680640 1 -- 172.21.15.7:0/2443296495 wait complete. 2024-03-19T01:23:40.972 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:23:26.868603+0000', 'modified': '2024-03-19T01:23:29.918429+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4266}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4266': {'gid': 4266, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.7:6835/1481596897', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6834', 'nonce': 1481596897}, {'type': 'v1', 'addr': '172.21.15.7:6835', 'nonce': 1481596897}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4266, 'qdb_cluster': [4266]} 2024-03-19T01:23:40.972 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-19T01:23:40.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-19T01:23:41.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.095+0000 7fa21680b640 1 Processor -- start 2024-03-19T01:23:41.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.095+0000 7fa21680b640 1 -- start start 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.095+0000 7fa21680b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100701a0 0x7fa21013f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.095+0000 7fa21680b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2100711c0 con 0x7fa210074390 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.095+0000 7fa21680b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa210074770 con 0x7fa210145e20 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.095+0000 7fa21680b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa210071430 con 0x7fa2100701a0 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.095+0000 7fa20ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa210074390 0x7fa21013eec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43364/0 (socket says 172.21.15.7:43364) 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.095+0000 7fa20ffff640 1 -- 172.21.15.7:0/2358594837 learned_addr learned my addr 172.21.15.7:0/2358594837 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa207fff640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100701a0 0x7fa21013f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 106879642 0 0) 0x7fa2100711c0 con 0x7fa210074390 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1fc003530 con 0x7fa210074390 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2868739829 0 0) 0x7fa1fc003530 con 0x7fa210074390 2024-03-19T01:23:41.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100701a0 msgr2=0x7fa21013f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100701a0 0x7fa21013f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 >> v1:172.21.15.167:6789/0 conn(0x7fa210145e20 legacy=0x7fa21014fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa210145340 con 0x7fa210074390 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fa200001680 con 0x7fa210074390 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 >> v1:172.21.15.7:6789/0 conn(0x7fa210074390 legacy=0x7fa21013eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1fc0039f0 0x7fa1fc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1fc004810 0x7fa1fc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100701a0 0x7fa1fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa210071430 con 0x7fa1fc0039f0 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa1fc00ff10 con 0x7fa1fc004810 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa1fc010110 con 0x7fa2100701a0 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20ffff640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1fc0039f0 0x7fa1fc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa207fff640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1fc004810 0x7fa1fc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa214da8640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100701a0 0x7fa1fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20ffff640 1 -- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100701a0 msgr2=0x7fa1fc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20ffff640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100701a0 0x7fa1fc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20ffff640 1 -- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1fc004810 msgr2=0x7fa1fc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20ffff640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1fc004810 0x7fa1fc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.099+0000 7fa20ffff640 1 -- 172.21.15.7:0/2358594837 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1fc003530 con 0x7fa1fc0039f0 2024-03-19T01:23:41.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa20ffff640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1fc0039f0 0x7fa1fc003e10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa20000c240 tx=0x7fa20001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99525e6e78244041 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200007d20 con 0x7fa1fc0039f0 2024-03-19T01:23:41.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2000a9070 con 0x7fa1fc0039f0 2024-03-19T01:23:41.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa20f7fe640 1 -- 172.21.15.7:0/2358594837 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200005d40 con 0x7fa1fc0039f0 2024-03-19T01:23:41.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 -- 172.21.15.7:0/2358594837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1fc0039f0 msgr2=0x7fa1fc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 --2- 172.21.15.7:0/2358594837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1fc0039f0 0x7fa1fc003e10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa20000c240 tx=0x7fa20001cc80 comp rx=0 tx=0).stop 2024-03-19T01:23:41.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa214da8640 1 -- 172.21.15.7:0/2358594837 reap_dead start 2024-03-19T01:23:41.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 -- 172.21.15.7:0/2358594837 shutdown_connections 2024-03-19T01:23:41.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 -- 172.21.15.7:0/2358594837 >> 172.21.15.7:0/2358594837 conn(0x7fa2100b9170 msgr2=0x7fa2100b9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:41.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 -- 172.21.15.7:0/2358594837 shutdown_connections 2024-03-19T01:23:41.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 -- 172.21.15.7:0/2358594837 wait complete. 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 Processor -- start 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 -- start start 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1fc0039f0 0x7fa210142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100b3950 0x7fa2100b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100b4540 0x7fa210137800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa21006fa00 con 0x7fa2100b3950 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa2100b7340 con 0x7fa1fc0039f0 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa21680b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa2100b0e00 con 0x7fa2100b4540 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa207fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100b3950 0x7fa2100b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa207fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100b3950 0x7fa2100b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40450/0 (socket says 172.21.15.7:40450) 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa207fff640 1 -- 172.21.15.7:0/3161334109 learned_addr learned my addr 172.21.15.7:0/3161334109 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.103+0000 7fa20ffff640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1fc0039f0 0x7fa210142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa214da8640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100b4540 0x7fa210137800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa207fff640 1 -- 172.21.15.7:0/3161334109 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100b4540 msgr2=0x7fa210137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa207fff640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100b4540 0x7fa210137800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa207fff640 1 -- 172.21.15.7:0/3161334109 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1fc0039f0 msgr2=0x7fa210142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa207fff640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1fc0039f0 0x7fa210142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa207fff640 1 -- 172.21.15.7:0/3161334109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2100b61f0 con 0x7fa2100b3950 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa207fff640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100b3950 0x7fa2100b4000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8011940 tx=0x7fa1f8011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa20d7fa640 1 -- 172.21.15.7:0/3161334109 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8019070 con 0x7fa2100b3950 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa20d7fa640 1 -- 172.21.15.7:0/3161334109 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f8004300 con 0x7fa2100b3950 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa21680b640 1 -- 172.21.15.7:0/3161334109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2100b49c0 con 0x7fa2100b3950 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa20d7fa640 1 -- 172.21.15.7:0/3161334109 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f800f9e0 con 0x7fa2100b3950 2024-03-19T01:23:41.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa21680b640 1 -- 172.21.15.7:0/3161334109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2100b6460 con 0x7fa2100b3950 2024-03-19T01:23:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa20d7fa640 1 -- 172.21.15.7:0/3161334109 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa1f801c070 con 0x7fa2100b3950 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa20d7fa640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa1f0067660 0x7fa1f0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa20d7fa640 1 -- 172.21.15.7:0/3161334109 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa1f8085d30 con 0x7fa2100b3950 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa20ffff640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa1f0067660 0x7fa1f0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa20ffff640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa1f0067660 0x7fa1f0069b20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa2000ab000 tx=0x7fa2000b0040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa21998d000 1 Processor -- start 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa21998d000 1 -- start start 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa21998d000 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 0x5635489031f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa21998d000 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x563548908280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.107+0000 7fa21998d000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548908ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5635487bf280 con 0x5635489087c0 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x563548727690 con 0x563548902e10 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x5635486aec50 con 0x563548903730 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548908ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548908ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40466/0 (socket says 172.21.15.7:40466) 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 -- 172.21.15.7:0/3211166234 learned_addr learned my addr 172.21.15.7:0/3211166234 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 -- 172.21.15.7:0/3211166234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 msgr2=0x563548908280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 --2- 172.21.15.7:0/3211166234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x563548908280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 -- 172.21.15.7:0/3211166234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 msgr2=0x5635489031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 --2- 172.21.15.7:0/3211166234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 0x5635489031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 --2- 172.21.15.7:0/3211166234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 0x5635489031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 -- 172.21.15.7:0/3211166234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635486aeea0 con 0x5635489087c0 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 --2- 172.21.15.7:0/3211166234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548908ba0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa21006f870 tx=0x7fa208007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa28e58e1d6144b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa205ffb640 1 -- 172.21.15.7:0/3211166234 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2080053a0 con 0x5635489087c0 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa205ffb640 1 -- 172.21.15.7:0/3211166234 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20801a070 con 0x5635489087c0 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa205ffb640 1 -- 172.21.15.7:0/3211166234 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208005740 con 0x5635489087c0 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- 172.21.15.7:0/3211166234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 msgr2=0x563548908ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 --2- 172.21.15.7:0/3211166234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548908ba0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa21006f870 tx=0x7fa208007f90 comp rx=0 tx=0).stop 2024-03-19T01:23:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- 172.21.15.7:0/3211166234 shutdown_connections 2024-03-19T01:23:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 --2- 172.21.15.7:0/3211166234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x563548908280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 --2- 172.21.15.7:0/3211166234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548908ba0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 --2- 172.21.15.7:0/3211166234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 0x5635489031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- 172.21.15.7:0/3211166234 >> 172.21.15.7:0/3211166234 conn(0x563548904030 msgr2=0x563548906420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- 172.21.15.7:0/3211166234 shutdown_connections 2024-03-19T01:23:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- 172.21.15.7:0/3211166234 wait complete. 2024-03-19T01:23:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- start start 2024-03-19T01:23:41.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 Processor -- start 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x5635488d7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x5635488d79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5635489087c0 0x5635488d7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x563548723730 con 0x563548902e10 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x5635486ef750 con 0x5635489087c0 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x56354890a120 con 0x563548903730 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 10 client.0 _ll_register_callbacks cb 0x5635488c3740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x5635488d7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x5635488d7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40468/0 (socket says 172.21.15.7:40468) 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 -- 172.21.15.7:0/1071550814 learned_addr learned my addr 172.21.15.7:0/1071550814 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa207fff640 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x5635488d79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa214da8640 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5635489087c0 0x5635488d7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 -- 172.21.15.7:0/1071550814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 msgr2=0x5635488d79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x5635488d79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 -- 172.21.15.7:0/1071550814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5635489087c0 msgr2=0x5635488d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5635489087c0 0x5635488d7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 -- 172.21.15.7:0/1071550814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635486af5b0 con 0x563548902e10 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa20ffff640 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x5635488d7490 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa20001cab0 tx=0x7fa200019a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd05179dbe928358 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa1eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa1eeffd640 1 -- 172.21.15.7:0/1071550814 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2000c7060 con 0x563548902e10 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa1eeffd640 1 -- 172.21.15.7:0/1071550814 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa2000a5760 con 0x563548902e10 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- 172.21.15.7:0/1071550814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa20801a070 con 0x563548902e10 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa1eeffd640 1 -- 172.21.15.7:0/1071550814 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fa2000cd050 con 0x563548902e10 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 10 client.4656 fetch_fsmap learned FSMap version 8 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.111+0000 7fa21998d000 1 -- 172.21.15.7:0/1071550814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5635487a88d0 con 0x563548902e10 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa1eeffd640 1 -- 172.21.15.7:0/1071550814 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20801a070 con 0x563548902e10 2024-03-19T01:23:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa1eeffd640 1 -- 172.21.15.7:0/1071550814 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 8) ==== 2284+0+0 (secure 0 0 0) 0x7fa2000d0020 con 0x563548902e10 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 10 client.4656 fetch_fsmap finished waiting for FSMap version 8 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 10 client.4656 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 20 client.4656 populate_metadata read hostname 'smithi007' 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] conn(0x56354891b660 0x56354891da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 4 client.4656 mds_command: new command op to 4266 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 -- 172.21.15.7:0/1071550814 --> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635486c2bc0 con 0x56354891b660 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa207fff640 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] conn(0x56354891b660 0x56354891da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa1eeffd640 10 client.4656 ms_handle_connect on v2:172.21.15.7:6834/1481596897 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa207fff640 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] conn(0x56354891b660 0x56354891da50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a861d55cbc7d9b8 server_cookie=84512be148ed254e in_seq=0 out_seq=0 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa1eeffd640 1 -- 172.21.15.7:0/1071550814 <== mds.0 v2:172.21.15.7:6834/1481596897 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5635486c2bc0 con 0x56354891b660 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa1eeffd640 10 client.4656 handle_command_reply: tid=0 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 client.4656 shutdown 2024-03-19T01:23:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 -- 172.21.15.7:0/1071550814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 msgr2=0x5635488d7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x5635488d7490 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa20001cab0 tx=0x7fa200019a20 comp rx=0 tx=0).stop 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 -- 172.21.15.7:0/1071550814 shutdown_connections 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] conn(0x56354891b660 0x56354891da50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x5635488d79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5635489087c0 0x5635488d7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 --2- 172.21.15.7:0/1071550814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x5635488d7490 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa1eeffd640 0 client.4656 ms_handle_reset on v2:172.21.15.7:6834/1481596897 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 -- 172.21.15.7:0/1071550814 >> 172.21.15.7:0/1071550814 conn(0x563548904030 msgr2=0x563548906420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 -- 172.21.15.7:0/1071550814 shutdown_connections 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 1 -- 172.21.15.7:0/1071550814 wait complete. 2024-03-19T01:23:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.115+0000 7fa21998d000 20 client.4656 trim_cache size 0 max 16384 2024-03-19T01:23:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.151+0000 7fa21998d000 1 Processor -- start 2024-03-19T01:23:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.151+0000 7fa21998d000 1 -- start start 2024-03-19T01:23:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.151+0000 7fa21998d000 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 0x5635488d4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x5635489758f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548975e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa2101e9150 con 0x5635489087c0 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa2101e82e0 con 0x563548902e10 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x5635487e2da0 con 0x563548903730 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa214da8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548975e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa214da8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548975e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40480/0 (socket says 172.21.15.7:40480) 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa214da8640 1 -- 172.21.15.7:0/694151476 learned_addr learned my addr 172.21.15.7:0/694151476 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa207fff640 1 --2- 172.21.15.7:0/694151476 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x5635489758f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa214da8640 1 -- 172.21.15.7:0/694151476 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 msgr2=0x5635489758f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 --2- 172.21.15.7:0/694151476 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 0x5635488d4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa214da8640 1 --2- 172.21.15.7:0/694151476 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x5635489758f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa214da8640 1 -- 172.21.15.7:0/694151476 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 msgr2=0x5635488d4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa214da8640 1 --2- 172.21.15.7:0/694151476 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 0x5635488d4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa214da8640 1 -- 172.21.15.7:0/694151476 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56354877f300 con 0x5635489087c0 2024-03-19T01:23:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa214da8640 1 --2- 172.21.15.7:0/694151476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548975e30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa208017040 tx=0x7fa20800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a331ec23b85ced8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa2057fa640 1 -- 172.21.15.7:0/694151476 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208024030 con 0x5635489087c0 2024-03-19T01:23:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa2057fa640 1 -- 172.21.15.7:0/694151476 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20801a070 con 0x5635489087c0 2024-03-19T01:23:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa2057fa640 1 -- 172.21.15.7:0/694151476 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208006b60 con 0x5635489087c0 2024-03-19T01:23:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- 172.21.15.7:0/694151476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 msgr2=0x563548975e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 --2- 172.21.15.7:0/694151476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548975e30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa208017040 tx=0x7fa20800f000 comp rx=0 tx=0).stop 2024-03-19T01:23:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- 172.21.15.7:0/694151476 shutdown_connections 2024-03-19T01:23:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 --2- 172.21.15.7:0/694151476 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x5635489758f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 --2- 172.21.15.7:0/694151476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635489087c0 0x563548975e30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 --2- 172.21.15.7:0/694151476 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563548902e10 0x5635488d4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- 172.21.15.7:0/694151476 >> 172.21.15.7:0/694151476 conn(0x563548904030 msgr2=0x563548920ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- 172.21.15.7:0/694151476 shutdown_connections 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- 172.21.15.7:0/694151476 wait complete. 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- start start 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 Processor -- start 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x563548976940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x563548976e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5635489087c0 0x5635489773c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5635487269a0 con 0x563548902e10 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x5635486bf7b0 con 0x5635489087c0 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x56354872bcc0 con 0x563548903730 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 10 client.0 _ll_register_callbacks cb 0x7fa1fc003700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x563548976940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x563548976940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40496/0 (socket says 172.21.15.7:40496) 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 -- 172.21.15.7:0/4122164770 learned_addr learned my addr 172.21.15.7:0/4122164770 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 -- 172.21.15.7:0/4122164770 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 msgr2=0x563548976e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa207fff640 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x563548976e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x563548976e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 -- 172.21.15.7:0/4122164770 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5635489087c0 msgr2=0x5635489773c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5635489087c0 0x5635489773c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 -- 172.21.15.7:0/4122164770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635486d7190 con 0x563548902e10 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa20ffff640 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x563548976940 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa20001d160 tx=0x7fa200007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a39d29e8bb0095 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa1eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa1eeffd640 1 -- 172.21.15.7:0/4122164770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200007de0 con 0x563548902e10 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa1eeffd640 1 -- 172.21.15.7:0/4122164770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa200004e00 con 0x563548902e10 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.155+0000 7fa21998d000 1 -- 172.21.15.7:0/4122164770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa20801a070 con 0x563548902e10 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa1eeffd640 1 -- 172.21.15.7:0/4122164770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fa200007040 con 0x563548902e10 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 10 client.4662 fetch_fsmap learned FSMap version 8 2024-03-19T01:23:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 -- 172.21.15.7:0/4122164770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5635486bb4b0 con 0x563548902e10 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa1eeffd640 1 -- 172.21.15.7:0/4122164770 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20801a070 con 0x563548902e10 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa1eeffd640 1 -- 172.21.15.7:0/4122164770 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 8) ==== 2284+0+0 (secure 0 0 0) 0x7fa2000d0040 con 0x563548902e10 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 10 client.4662 fetch_fsmap finished waiting for FSMap version 8 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 10 client.4662 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 20 client.4662 populate_metadata read hostname 'smithi007' 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] conn(0x563548975720 0x563548975b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 4 client.4662 mds_command: new command op to 4266 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 -- 172.21.15.7:0/4122164770 --> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] -- command(tid 0: {"prefix": "status"}) -- 0x5635488d6040 con 0x563548975720 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa207fff640 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] conn(0x563548975720 0x563548975b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa1eeffd640 10 client.4662 ms_handle_connect on v2:172.21.15.7:6834/1481596897 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa207fff640 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] conn(0x563548975720 0x563548975b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=144521e1bc52c7c6 server_cookie=56d3a59d66eed7b0 in_seq=0 out_seq=0 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa1eeffd640 1 -- 172.21.15.7:0/4122164770 <== mds.0 v2:172.21.15.7:6834/1481596897 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x5635488d6040 con 0x563548975720 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa1eeffd640 10 client.4662 handle_command_reply: tid=0 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 client.4662 shutdown 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 -- 172.21.15.7:0/4122164770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 msgr2=0x563548976940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x563548976940 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa20001d160 tx=0x7fa200007900 comp rx=0 tx=0).stop 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 -- 172.21.15.7:0/4122164770 shutdown_connections 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.7:6834/1481596897,v1:172.21.15.7:6835/1481596897] conn(0x563548975720 0x563548975b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563548903730 0x563548976e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5635489087c0 0x5635489773c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa1eeffd640 0 client.4662 ms_handle_reset on v2:172.21.15.7:6834/1481596897 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 --2- 172.21.15.7:0/4122164770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563548902e10 0x563548976940 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 -- 172.21.15.7:0/4122164770 >> 172.21.15.7:0/4122164770 conn(0x563548904030 msgr2=0x5635488d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 -- 172.21.15.7:0/4122164770 shutdown_connections 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 1 -- 172.21.15.7:0/4122164770 wait complete. 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21998d000 20 client.4662 trim_cache size 0 max 16384 2024-03-19T01:23:41.169 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "dafc8aaa-17d4-48ac-88de-2f029fdf817b", 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 4266, 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 13.672458020000001, 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 8, 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 21, 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 21, 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 20.806718478000001 2024-03-19T01:23:41.170 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:23:41.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 -- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa1f0067660 msgr2=0x7fa1f0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa1f0067660 0x7fa1f0069b20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa2000ab000 tx=0x7fa2000b0040 comp rx=0 tx=0).stop 2024-03-19T01:23:41.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 -- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100b3950 msgr2=0x7fa2100b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100b3950 0x7fa2100b4000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8011940 tx=0x7fa1f8011e00 comp rx=0 tx=0).stop 2024-03-19T01:23:41.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 -- 172.21.15.7:0/3161334109 shutdown_connections 2024-03-19T01:23:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa1f0067660 0x7fa1f0069b20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2100b4540 0x7fa210137800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100b3950 0x7fa2100b4000 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 --2- 172.21.15.7:0/3161334109 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1fc0039f0 0x7fa210142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 -- 172.21.15.7:0/3161334109 >> 172.21.15.7:0/3161334109 conn(0x7fa2100b9170 msgr2=0x7fa210143950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 -- 172.21.15.7:0/3161334109 shutdown_connections 2024-03-19T01:23:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.159+0000 7fa21680b640 1 -- 172.21.15.7:0/3161334109 wait complete. 2024-03-19T01:23:41.181 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-19T01:23:41.207 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-19T01:23:41.208 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-19T01:23:41.208 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-19T01:23:41.208 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T01:23:41.208 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T01:23:41.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:41.208 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-03-19T01:23:41.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:41.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2024-03-19T01:23:41.266 INFO:teuthology.orchestra.run.smithi007.stderr:Cannot find device "ceph-brx" 2024-03-19T01:23:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:23:41.267 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-19T01:23:41.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-19T01:23:41.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f906e5cc640 1 Processor -- start 2024-03-19T01:23:41.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f906e5cc640 1 -- start start 2024-03-19T01:23:41.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f906e5cc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f906806df50 0x7f906806e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f906e5cc640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f9068147d10 0x7f9068148170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f906e5cc640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90680aec10 con 0x7f90680af510 2024-03-19T01:23:41.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f906e5cc640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f906806be90 con 0x7f9068147d10 2024-03-19T01:23:41.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f906e5cc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f906809f440 con 0x7f906806df50 2024-03-19T01:23:41.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f9067fff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f906806df50 0x7f906806e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f90677fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f9068147d10 0x7f9068148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f9067fff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f906806df50 0x7f906806e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:55594/0 (socket says 172.21.15.7:55594) 2024-03-19T01:23:41.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f9067fff640 1 -- 172.21.15.7:0/2461849208 learned_addr learned my addr 172.21.15.7:0/2461849208 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:41.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f9067fff640 1 -- 172.21.15.7:0/2461849208 >> v2:172.21.15.167:3300/0 conn(0x7f9068147d10 msgr2=0x7f9068148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:41.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f9067fff640 1 --2- 172.21.15.7:0/2461849208 >> v2:172.21.15.167:3300/0 conn(0x7f9068147d10 0x7f9068148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f9067fff640 1 -- 172.21.15.7:0/2461849208 >> v1:172.21.15.167:6789/0 conn(0x7f90680af510 legacy=0x7f90680af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f9067fff640 1 -- 172.21.15.7:0/2461849208 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906806c0e0 con 0x7f906806df50 2024-03-19T01:23:41.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.423+0000 7f90677fe640 1 --2- 172.21.15.7:0/2461849208 >> v2:172.21.15.167:3300/0 conn(0x7f9068147d10 0x7f9068148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f9067fff640 1 --2- 172.21.15.7:0/2461849208 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f906806df50 0x7f906806e9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9058009870 tx=0x7f905801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=207b4833f995de00 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f9066ffd640 1 -- 172.21.15.7:0/2461849208 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90580b0070 con 0x7f906806df50 2024-03-19T01:23:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f9066ffd640 1 -- 172.21.15.7:0/2461849208 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9058004230 con 0x7f906806df50 2024-03-19T01:23:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f9066ffd640 1 -- 172.21.15.7:0/2461849208 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058003620 con 0x7f906806df50 2024-03-19T01:23:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f906e5cc640 1 -- 172.21.15.7:0/2461849208 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f906806df50 msgr2=0x7f906806e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f906e5cc640 1 --2- 172.21.15.7:0/2461849208 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f906806df50 0x7f906806e9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9058009870 tx=0x7f905801c440 comp rx=0 tx=0).stop 2024-03-19T01:23:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f906e5cc640 1 -- 172.21.15.7:0/2461849208 shutdown_connections 2024-03-19T01:23:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f906e5cc640 1 --2- 172.21.15.7:0/2461849208 >> v2:172.21.15.167:3300/0 conn(0x7f9068147d10 0x7f9068148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f906e5cc640 1 --2- 172.21.15.7:0/2461849208 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f906806df50 0x7f906806e9e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f906e5cc640 1 -- 172.21.15.7:0/2461849208 >> 172.21.15.7:0/2461849208 conn(0x7f90680b6510 msgr2=0x7f90680b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f906e5cc640 1 -- 172.21.15.7:0/2461849208 shutdown_connections 2024-03-19T01:23:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.427+0000 7f906e5cc640 1 -- 172.21.15.7:0/2461849208 wait complete. 2024-03-19T01:23:41.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 Processor -- start 2024-03-19T01:23:41.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 -- start start 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f906806df50 0x7f90680b3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90680af510 0x7f9068141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9068147d10 0x7f906813cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9068154730 con 0x7f906806df50 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f90681545b0 con 0x7f9068147d10 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f90681548b0 con 0x7f90680af510 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9067fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f906806df50 0x7f90680b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9067fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f906806df50 0x7f90680b3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40512/0 (socket says 172.21.15.7:40512) 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9067fff640 1 -- 172.21.15.7:0/3591339006 learned_addr learned my addr 172.21.15.7:0/3591339006 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906cb69640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9068147d10 0x7f906813cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f90677fe640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90680af510 0x7f9068141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9067fff640 1 -- 172.21.15.7:0/3591339006 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90680af510 msgr2=0x7f9068141240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9067fff640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90680af510 0x7f9068141240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9067fff640 1 -- 172.21.15.7:0/3591339006 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9068147d10 msgr2=0x7f906813cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9067fff640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9068147d10 0x7f906813cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9067fff640 1 -- 172.21.15.7:0/3591339006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90680b2c30 con 0x7f906806df50 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9067fff640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f906806df50 0x7f90680b3900 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f905801a070 tx=0x7f90580191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9064ff9640 1 -- 172.21.15.7:0/3591339006 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90580ba020 con 0x7f906806df50 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9064ff9640 1 -- 172.21.15.7:0/3591339006 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9058003de0 con 0x7f906806df50 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90680b10d0 con 0x7f906806df50 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f9064ff9640 1 -- 172.21.15.7:0/3591339006 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90580b0040 con 0x7f906806df50 2024-03-19T01:23:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90680b1610 con 0x7f906806df50 2024-03-19T01:23:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.431+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f906806c350 con 0x7f906806df50 2024-03-19T01:23:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.435+0000 7f9064ff9640 1 -- 172.21.15.7:0/3591339006 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f90580bd020 con 0x7f906806df50 2024-03-19T01:23:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.435+0000 7f9064ff9640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9054067690 0x7f9054069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.435+0000 7f9064ff9640 1 -- 172.21.15.7:0/3591339006 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f905811fa10 con 0x7f906806df50 2024-03-19T01:23:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.435+0000 7f90677fe640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9054067690 0x7f9054069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:41.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.435+0000 7f90677fe640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9054067690 0x7f9054069b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f906806eb90 tx=0x7f905c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:41.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.439+0000 7f9064ff9640 1 -- 172.21.15.7:0/3591339006 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f90580c4030 con 0x7f906806df50 2024-03-19T01:23:41.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.635+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9068075460 con 0x7f906806df50 2024-03-19T01:23:41.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.635+0000 7f9064ff9640 1 -- 172.21.15.7:0/3591339006 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f90680b10d0 con 0x7f906806df50 2024-03-19T01:23:41.642 INFO:teuthology.orchestra.run.smithi007.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-19T01:23:41.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9054067690 msgr2=0x7f9054069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9054067690 0x7f9054069b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f906806eb90 tx=0x7f905c008040 comp rx=0 tx=0).stop 2024-03-19T01:23:41.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f906806df50 msgr2=0x7f90680b3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:41.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f906806df50 0x7f90680b3900 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f905801a070 tx=0x7f90580191e0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 shutdown_connections 2024-03-19T01:23:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9054067690 0x7f9054069b50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90680af510 0x7f9068141240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9068147d10 0x7f906813cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 --2- 172.21.15.7:0/3591339006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f906806df50 0x7f90680b3900 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 >> 172.21.15.7:0/3591339006 conn(0x7f90680b6510 msgr2=0x7f90680b68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 shutdown_connections 2024-03-19T01:23:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41.639+0000 7f906e5cc640 1 -- 172.21.15.7:0/3591339006 wait complete. 2024-03-19T01:23:41.661 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-19T01:23:41.661 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T01:23:41.661 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi007.front.sepia.ceph.com 2024-03-19T01:23:41.661 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-19T01:23:41.662 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T01:23:41.662 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-19T01:23:41.662 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T01:23:41.662 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T01:23:41.662 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-19T01:23:41.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:41.662 DEBUG:teuthology.orchestra.run.smithi007:> ip addr 2024-03-19T01:23:41.668 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T01:23:41.668 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T01:23:41.668 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T01:23:41.668 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:23:41.668 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2024-03-19T01:23:41.668 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:23:41.668 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 42490sec preferred_lft 42490sec 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-19T01:23:41.669 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2024-03-19T01:23:41.670 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2024-03-19T01:23:41.670 INFO:teuthology.orchestra.run.smithi007.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-19T01:23:41.670 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 02:42:33:01:ef:e0 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:23:41.670 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-19T01:23:41.670 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:23:41.670 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T01:23:41.671 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:23:41.671 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-19T01:23:41.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add name ceph-brx type bridge 2024-03-19T01:23:41.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr flush dev ceph-brx 2024-03-19T01:23:41.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set ceph-brx up 2024-03-19T01:23:41.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T01:23:41.671 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-19T01:23:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:23:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:23:41.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:41.805 DEBUG:teuthology.orchestra.run.smithi007:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T01:23:41.814 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-03-19T01:23:41.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:41.815 DEBUG:teuthology.orchestra.run.smithi007:> route 2024-03-19T01:23:41.882 INFO:teuthology.orchestra.run.smithi007.stdout:Kernel IP routing table 2024-03-19T01:23:41.882 INFO:teuthology.orchestra.run.smithi007.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T01:23:41.882 INFO:teuthology.orchestra.run.smithi007.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-19T01:23:41.883 INFO:teuthology.orchestra.run.smithi007.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-19T01:23:41.883 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-19T01:23:41.883 INFO:teuthology.orchestra.run.smithi007.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T01:23:41.883 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:23:41.883 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-19T01:23:41.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-19T01:23:41.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-19T01:23:41.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-19T01:23:41.883 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-19T01:23:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:23:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:23:41.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:41.997 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-03-19T01:23:42.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:42.047 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list-id 2024-03-19T01:23:42.096 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:23:42.096 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-19T01:23:42.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:23:42.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-19T01:23:42.096 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-19T01:23:42.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:23:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:23:42.210 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-19T01:23:42.210 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:23:42.210 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-19T01:23:42.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-19T01:23:42.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T01:23:42.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-19T01:23:42.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-19T01:23:42.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-19T01:23:42.211 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-19T01:23:42.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:23:42.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:23:42.303 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:23:42.303 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-19T01:23:42.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set brx.0 up 2024-03-19T01:23:42.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T01:23:42.304 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-19T01:23:42.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:23:42.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:23:42.406 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-19T01:23:42.407 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T01:23:42.407 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-19T01:23:42.459 INFO:teuthology.orchestra.run.smithi007.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-19T01:23:42.460 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T01:23:42.460 DEBUG:teuthology.orchestra.run.smithi007:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-19T01:23:42.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-03-19T01:23:42.562 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T01:23:42.609 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-03-19T01:23:42.609 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/run/lock 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-03-19T01:23:42.610 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/fs/nfsd 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/run/rpc_pipefs 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-0 2024-03-19T01:23:42.611 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-1 2024-03-19T01:23:42.612 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-2 2024-03-19T01:23:42.612 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-3 2024-03-19T01:23:42.612 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-03-19T01:23:42.612 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:23:42.612 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:23:42.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:42.612 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-03-19T01:23:42.655 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-19T01:23:42.656 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-19T01:23:42.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-03-19T01:23:42.713 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T01:23:42.759 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-03-19T01:23:42.759 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-03-19T01:23:42.759 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-03-19T01:23:42.759 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-03-19T01:23:42.759 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-03-19T01:23:42.759 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-03-19T01:23:42.759 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-03-19T01:23:42.759 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-03-19T01:23:42.759 INFO:teuthology.orchestra.run.smithi007.stdout:/run/lock 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/fs/nfsd 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/run/rpc_pipefs 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-03-19T01:23:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-03-19T01:23:42.761 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-0 2024-03-19T01:23:42.761 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-1 2024-03-19T01:23:42.761 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-2 2024-03-19T01:23:42.761 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-3 2024-03-19T01:23:42.761 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-03-19T01:23:42.761 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:23:42.761 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:23:42.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:42.762 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-03-19T01:23:42.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:2024-03-19T01:23:42.779+0000 7f09b0720580 -1 init, newargv = 0x559c158cd940 newargc=15 2024-03-19T01:23:42.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:ceph-fuse[34238]: starting ceph client 2024-03-19T01:23:42.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:ceph-fuse[34238]: starting fuse 2024-03-19T01:23:43.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-03-19T01:23:43.905 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T01:23:43.952 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-03-19T01:23:43.952 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/run/lock 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-03-19T01:23:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/fs/nfsd 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/run/rpc_pipefs 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-0 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-1 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-2 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-3 2024-03-19T01:23:43.954 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-03-19T01:23:43.955 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:23:43.955 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:23:43.955 INFO:teuthology.orchestra.run.smithi007.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-19T01:23:43.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:43.956 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-03-19T01:23:44.007 INFO:teuthology.orchestra.run.smithi007.stdout:45 2024-03-19T01:23:44.007 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo stdin-killer -- python3 -c ' 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> import glob 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> import re 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> import os 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> import subprocess 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> def _find_admin_socket(client_name): 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> files = glob.glob(asok_path) 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> # Given a non-glob path, it better be there 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> if "*" not in asok_path: 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> assert(len(files) == 1) 2024-03-19T01:23:44.008 DEBUG:teuthology.orchestra.run.smithi007:> return files[0] 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> for f in files: 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> contents = proc_f.read() 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> if mountpoint in contents: 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> return f 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> print(_find_admin_socket("client.0")) 2024-03-19T01:23:44.009 DEBUG:teuthology.orchestra.run.smithi007:> ' 2024-03-19T01:23:44.137 INFO:teuthology.orchestra.run.smithi007.stdout:/var/run/ceph/ceph-client.0.34238.asok 2024-03-19T01:23:44.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:23:44.153 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.34238.asok 2024-03-19T01:23:44.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:44.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.34238.asok status 2024-03-19T01:23:44.326 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:23:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "metadata": { 2024-03-19T01:23:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-19T01:23:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T01:23:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "entity_id": "0", 2024-03-19T01:23:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": "smithi007", 2024-03-19T01:23:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-19T01:23:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "pid": "34238", 2024-03-19T01:23:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "root": "/" 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_count": 0, 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_pinned_count": 0, 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 4671, 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "inst": { 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "name": { 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "client", 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "num": 4671 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 755299148 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:23:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 755299148 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "inst_str": "client.4671 192.168.0.1:0/755299148", 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "addr_str": "192.168.0.1:0/755299148", 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "inode_count": 1, 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_epoch": 8, 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch": 21, 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch_barrier": 0, 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklisted": false, 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs" 2024-03-19T01:23:44.328 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:23:44.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:44.341 DEBUG:teuthology.orchestra.run.smithi007:> stat --file-system '--printf=%T 2024-03-19T01:23:44.341 DEBUG:teuthology.orchestra.run.smithi007:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-19T01:23:44.395 INFO:teuthology.orchestra.run.smithi007.stdout:fuseblk 2024-03-19T01:23:44.396 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-19T01:23:44.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:44.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-19T01:23:44.458 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-19T01:23:44.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.635+0000 7f4b88186640 1 Processor -- start 2024-03-19T01:23:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.635+0000 7f4b88186640 1 -- start start 2024-03-19T01:23:44.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b88186640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b80074390 0x7f4b800b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b88186640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b800b32e0 0x7f4b801456b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b88186640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b800b0c10 con 0x7f4b80145bf0 2024-03-19T01:23:44.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b88186640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f4b80074770 con 0x7f4b800b32e0 2024-03-19T01:23:44.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b88186640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4b800b17d0 con 0x7f4b80074390 2024-03-19T01:23:44.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b85721640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b800b32e0 0x7f4b801456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b85721640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b800b32e0 0x7f4b801456b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42748/0 (socket says 172.21.15.7:42748) 2024-03-19T01:23:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b85721640 1 -- 172.21.15.7:0/2683940366 learned_addr learned my addr 172.21.15.7:0/2683940366 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b85f22640 1 --2- 172.21.15.7:0/2683940366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b80074390 0x7f4b800b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b85721640 1 -- 172.21.15.7:0/2683940366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b80074390 msgr2=0x7f4b800b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b85721640 1 --2- 172.21.15.7:0/2683940366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b80074390 0x7f4b800b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b85721640 1 -- 172.21.15.7:0/2683940366 >> v1:172.21.15.7:6789/0 conn(0x7f4b80145bf0 legacy=0x7f4b8014b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b85721640 1 -- 172.21.15.7:0/2683940366 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b800b2e50 con 0x7f4b800b32e0 2024-03-19T01:23:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b85721640 1 --2- 172.21.15.7:0/2683940366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b800b32e0 0x7f4b801456b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009870 tx=0x7f4b7c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d482b3e0cfd78c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b84f20640 1 -- 172.21.15.7:0/2683940366 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0b0070 con 0x7f4b800b32e0 2024-03-19T01:23:44.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b84f20640 1 -- 172.21.15.7:0/2683940366 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b7c0ab600 con 0x7f4b800b32e0 2024-03-19T01:23:44.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.639+0000 7f4b84f20640 1 -- 172.21.15.7:0/2683940366 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0ab9d0 con 0x7f4b800b32e0 2024-03-19T01:23:44.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- 172.21.15.7:0/2683940366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b800b32e0 msgr2=0x7f4b801456b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 --2- 172.21.15.7:0/2683940366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b800b32e0 0x7f4b801456b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009870 tx=0x7f4b7c01c4d0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- 172.21.15.7:0/2683940366 shutdown_connections 2024-03-19T01:23:44.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 --2- 172.21.15.7:0/2683940366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b800b32e0 0x7f4b801456b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 --2- 172.21.15.7:0/2683940366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b80074390 0x7f4b800b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- 172.21.15.7:0/2683940366 >> 172.21.15.7:0/2683940366 conn(0x7f4b800b5bf0 msgr2=0x7f4b800b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:44.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- 172.21.15.7:0/2683940366 shutdown_connections 2024-03-19T01:23:44.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- 172.21.15.7:0/2683940366 wait complete. 2024-03-19T01:23:44.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 Processor -- start 2024-03-19T01:23:44.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- start start 2024-03-19T01:23:44.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b80074390 0x7f4b8013b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b800b32e0 0x7f4b80146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b80145bf0 0x7f4b801e4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4b80158ba0 con 0x7f4b80074390 2024-03-19T01:23:44.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4b80146000 con 0x7f4b80145bf0 2024-03-19T01:23:44.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4b80158d20 con 0x7f4b800b32e0 2024-03-19T01:23:44.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85f22640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b80074390 0x7f4b8013b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85f22640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b80074390 0x7f4b8013b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42750/0 (socket says 172.21.15.7:42750) 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85f22640 1 -- 172.21.15.7:0/3585450670 learned_addr learned my addr 172.21.15.7:0/3585450670 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85721640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b800b32e0 0x7f4b80146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b86723640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b80145bf0 0x7f4b801e4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85f22640 1 -- 172.21.15.7:0/3585450670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b800b32e0 msgr2=0x7f4b80146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85f22640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b800b32e0 0x7f4b80146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85f22640 1 -- 172.21.15.7:0/3585450670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b80145bf0 msgr2=0x7f4b801e4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85f22640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b80145bf0 0x7f4b801e4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85f22640 1 -- 172.21.15.7:0/3585450670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b801e7bd0 con 0x7f4b80074390 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b85f22640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b80074390 0x7f4b8013b8c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000d800 tx=0x7f4b7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b767fc640 1 -- 172.21.15.7:0/3585450670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7000b6c0 con 0x7f4b80074390 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b767fc640 1 -- 172.21.15.7:0/3585450670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b70019070 con 0x7f4b80074390 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b801e7eb0 con 0x7f4b80074390 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b767fc640 1 -- 172.21.15.7:0/3585450670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7000ba60 con 0x7f4b80074390 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.643+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b800b0540 con 0x7f4b80074390 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.647+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b80075460 con 0x7f4b80074390 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.647+0000 7f4b767fc640 1 -- 172.21.15.7:0/3585450670 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4b7000bc00 con 0x7f4b80074390 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.647+0000 7f4b767fc640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b64067660 0x7f4b64069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.647+0000 7f4b767fc640 1 -- 172.21.15.7:0/3585450670 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4b70085c40 con 0x7f4b80074390 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.647+0000 7f4b85721640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b64067660 0x7f4b64069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.647+0000 7f4b85721640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b64067660 0x7f4b64069b20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0ad000 tx=0x7f4b7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:44.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.651+0000 7f4b767fc640 1 -- 172.21.15.7:0/3585450670 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4b70056950 con 0x7f4b80074390 2024-03-19T01:23:44.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.811+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b80134760 con 0x7f4b80074390 2024-03-19T01:23:44.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.815+0000 7f4b767fc640 1 -- 172.21.15.7:0/3585450670 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7f4b70056250 con 0x7f4b80074390 2024-03-19T01:23:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:44.818 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:44.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b64067660 msgr2=0x7f4b64069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b64067660 0x7f4b64069b20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0ad000 tx=0x7f4b7c008040 comp rx=0 tx=0).stop 2024-03-19T01:23:44.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b80074390 msgr2=0x7f4b8013b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b80074390 0x7f4b8013b8c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000d800 tx=0x7f4b7000dcc0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 shutdown_connections 2024-03-19T01:23:44.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b64067660 0x7f4b64069b20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b800b32e0 0x7f4b80146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b80145bf0 0x7f4b801e4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 --2- 172.21.15.7:0/3585450670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b80074390 0x7f4b8013b8c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 >> 172.21.15.7:0/3585450670 conn(0x7f4b800b5bf0 msgr2=0x7f4b80067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:44.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 shutdown_connections 2024-03-19T01:23:44.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.819+0000 7f4b88186640 1 -- 172.21.15.7:0/3585450670 wait complete. 2024-03-19T01:23:44.837 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:44.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:44.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961cc76640 1 Processor -- start 2024-03-19T01:23:44.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961cc76640 1 -- start start 2024-03-19T01:23:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961cc76640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 0x7f9618147880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961cc76640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96180b6280 con 0x7f9618147dc0 2024-03-19T01:23:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961cc76640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96180af9d0 con 0x7f961806b520 2024-03-19T01:23:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961cc76640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f961806b900 con 0x7f96180b4ce0 2024-03-19T01:23:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9616d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9618147dc0 0x7f96181520e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35034/0 (socket says 172.21.15.7:35034) 2024-03-19T01:23:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9616d9d640 1 -- 172.21.15.7:0/2533968555 learned_addr learned my addr 172.21.15.7:0/2533968555 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9615d9b640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 0x7f9618147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3616838671 0 0) 0x7f96080014a0 con 0x7f9618147dc0 2024-03-19T01:23:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9604003530 con 0x7f9618147dc0 2024-03-19T01:23:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 949247283 0 0) 0x7f9604003530 con 0x7f9618147dc0 2024-03-19T01:23:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 msgr2=0x7f9618147880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 0x7f9618147880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 >> v1:172.21.15.167:6789/0 conn(0x7f961806b520 legacy=0x7f96180af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96180b54e0 con 0x7f9618147dc0 2024-03-19T01:23:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f9608001bd0 con 0x7f9618147dc0 2024-03-19T01:23:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9615d9b640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 0x7f9618147880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 >> v1:172.21.15.7:6789/0 conn(0x7f9618147dc0 legacy=0x7f96181520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96040039f0 0x7f9604003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 0x7f9604004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9604004d50 0x7f96040051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f961806b900 con 0x7f9604004d50 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f960400bb60 con 0x7f96040039f0 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f960400bd30 con 0x7f96180b4ce0 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9616d9d640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9604004d50 0x7f96040051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9615d9b640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 0x7f9604004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961659c640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96040039f0 0x7f9604003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9616d9d640 1 -- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 msgr2=0x7f9604004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9616d9d640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 0x7f9604004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9616d9d640 1 -- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96040039f0 msgr2=0x7f9604003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9616d9d640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96040039f0 0x7f9604003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9616d9d640 1 -- 172.21.15.7:0/2533968555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9604003530 con 0x7f9604004d50 2024-03-19T01:23:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f9615d9b640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96180b4ce0 0x7f9604004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.963+0000 7f961659c640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96040039f0 0x7f9604003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f9616d9d640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9604004d50 0x7f96040051d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f960800a960 tx=0x7f960801d240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba5afefe6ef68a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960800a580 con 0x7f9604004d50 2024-03-19T01:23:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96080ad630 con 0x7f9604004d50 2024-03-19T01:23:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961559a640 1 -- 172.21.15.7:0/2533968555 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96080ada40 con 0x7f9604004d50 2024-03-19T01:23:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961cc76640 1 -- 172.21.15.7:0/2533968555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9604004d50 msgr2=0x7f96040051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961cc76640 1 --2- 172.21.15.7:0/2533968555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9604004d50 0x7f96040051d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f960800a960 tx=0x7f960801d240 comp rx=0 tx=0).stop 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f9616d9d640 1 -- 172.21.15.7:0/2533968555 reap_dead start 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961cc76640 1 -- 172.21.15.7:0/2533968555 shutdown_connections 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961cc76640 1 -- 172.21.15.7:0/2533968555 >> 172.21.15.7:0/2533968555 conn(0x7f961806e1b0 msgr2=0x7f961806e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961cc76640 1 -- 172.21.15.7:0/2533968555 shutdown_connections 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961cc76640 1 -- 172.21.15.7:0/2533968555 wait complete. 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961cc76640 1 Processor -- start 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.967+0000 7f961cc76640 1 -- start start 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961cc76640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96181cefe0 0x7f96181dc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961cc76640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96181d0f50 0x7f96181dff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961cc76640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96181e04d0 0x7f96181e0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961cc76640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f961809fc30 con 0x7f96181d0f50 2024-03-19T01:23:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961cc76640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f96180b5f00 con 0x7f96181cefe0 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961cc76640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f96181d0ac0 con 0x7f96181e04d0 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9615d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96181d0f50 0x7f96181dff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9615d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96181d0f50 0x7f96181dff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42768/0 (socket says 172.21.15.7:42768) 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9615d9b640 1 -- 172.21.15.7:0/833256292 learned_addr learned my addr 172.21.15.7:0/833256292 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9616d9d640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96181e04d0 0x7f96181e0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961659c640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96181cefe0 0x7f96181dc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9615d9b640 1 -- 172.21.15.7:0/833256292 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96181e04d0 msgr2=0x7f96181e0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9615d9b640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96181e04d0 0x7f96181e0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9615d9b640 1 -- 172.21.15.7:0/833256292 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96181cefe0 msgr2=0x7f96181dc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9615d9b640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96181cefe0 0x7f96181dc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9615d9b640 1 -- 172.21.15.7:0/833256292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96181cf3c0 con 0x7f96181d0f50 2024-03-19T01:23:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f9615d9b640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96181d0f50 0x7f96181dff90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f96000116a0 tx=0x7f9600011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f960effd640 1 -- 172.21.15.7:0/833256292 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960000f960 con 0x7f96181d0f50 2024-03-19T01:23:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f960effd640 1 -- 172.21.15.7:0/833256292 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f960000fb00 con 0x7f96181d0f50 2024-03-19T01:23:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96181d1730 con 0x7f96181d0f50 2024-03-19T01:23:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f960effd640 1 -- 172.21.15.7:0/833256292 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600015460 con 0x7f96181d0f50 2024-03-19T01:23:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96180b1270 con 0x7f96181d0f50 2024-03-19T01:23:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9618075570 con 0x7f96181d0f50 2024-03-19T01:23:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f960effd640 1 -- 172.21.15.7:0/833256292 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9600015600 con 0x7f96181d0f50 2024-03-19T01:23:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f960effd640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f95f8067660 0x7f95f8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f960effd640 1 -- 172.21.15.7:0/833256292 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9600086460 con 0x7f96181d0f50 2024-03-19T01:23:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961659c640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f95f8067660 0x7f95f8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.971+0000 7f961659c640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f95f8067660 0x7f95f8069b20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9610012370 tx=0x7f961000aa40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:44.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:44.975+0000 7f960effd640 1 -- 172.21.15.7:0/833256292 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9600019030 con 0x7f96181d0f50 2024-03-19T01:23:45.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.135+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f96180b2180 con 0x7f96181d0f50 2024-03-19T01:23:45.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.135+0000 7f960effd640 1 -- 172.21.15.7:0/833256292 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f9600057250 con 0x7f96181d0f50 2024-03-19T01:23:45.137 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:45.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f95f8067660 msgr2=0x7f95f8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f95f8067660 0x7f95f8069b20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9610012370 tx=0x7f961000aa40 comp rx=0 tx=0).stop 2024-03-19T01:23:45.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96181d0f50 msgr2=0x7f96181dff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96181d0f50 0x7f96181dff90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f96000116a0 tx=0x7f9600011b60 comp rx=0 tx=0).stop 2024-03-19T01:23:45.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 shutdown_connections 2024-03-19T01:23:45.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f95f8067660 0x7f95f8069b20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96181e04d0 0x7f96181e0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96181d0f50 0x7f96181dff90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 --2- 172.21.15.7:0/833256292 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96181cefe0 0x7f96181dc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 >> 172.21.15.7:0/833256292 conn(0x7f961806e1b0 msgr2=0x7f961806e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:45.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 shutdown_connections 2024-03-19T01:23:45.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.139+0000 7f961cc76640 1 -- 172.21.15.7:0/833256292 wait complete. 2024-03-19T01:23:45.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:45.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.275+0000 7ff83ac9c640 1 Processor -- start 2024-03-19T01:23:45.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.275+0000 7ff83ac9c640 1 -- start start 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff83ac9c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff834074390 0x7ff834145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff83ac9c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff834142a60 0x7ff834145940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff83ac9c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff834142e70 con 0x7ff8340b3070 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff83ac9c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7ff834145e80 con 0x7ff834074390 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff83ac9c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff834074770 con 0x7ff834142a60 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff838a38640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff834074390 0x7ff834145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff838a38640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff834074390 0x7ff834145400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42774/0 (socket says 172.21.15.7:42774) 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff838a38640 1 -- 172.21.15.7:0/424911026 learned_addr learned my addr 172.21.15.7:0/424911026 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff839239640 1 --2- 172.21.15.7:0/424911026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff834142a60 0x7ff834145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff838a38640 1 -- 172.21.15.7:0/424911026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff834142a60 msgr2=0x7ff834145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff838a38640 1 --2- 172.21.15.7:0/424911026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff834142a60 0x7ff834145940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff838a38640 1 -- 172.21.15.7:0/424911026 >> v1:172.21.15.167:6789/0 conn(0x7ff8340b3070 legacy=0x7ff834142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:45.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff838a38640 1 -- 172.21.15.7:0/424911026 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff834071110 con 0x7ff834074390 2024-03-19T01:23:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff838a38640 1 --2- 172.21.15.7:0/424911026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff834074390 0x7ff834145400 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff820009870 tx=0x7ff82001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a37ab0b50b147f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff8337fe640 1 -- 172.21.15.7:0/424911026 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8200b1070 con 0x7ff834074390 2024-03-19T01:23:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff8337fe640 1 -- 172.21.15.7:0/424911026 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8200ac630 con 0x7ff834074390 2024-03-19T01:23:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.279+0000 7ff8337fe640 1 -- 172.21.15.7:0/424911026 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8200a8050 con 0x7ff834074390 2024-03-19T01:23:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- 172.21.15.7:0/424911026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff834074390 msgr2=0x7ff834145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 --2- 172.21.15.7:0/424911026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff834074390 0x7ff834145400 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff820009870 tx=0x7ff82001c520 comp rx=0 tx=0).stop 2024-03-19T01:23:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- 172.21.15.7:0/424911026 shutdown_connections 2024-03-19T01:23:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 --2- 172.21.15.7:0/424911026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff834142a60 0x7ff834145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 --2- 172.21.15.7:0/424911026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff834074390 0x7ff834145400 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- 172.21.15.7:0/424911026 >> 172.21.15.7:0/424911026 conn(0x7ff8340b9120 msgr2=0x7ff8340b9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- 172.21.15.7:0/424911026 shutdown_connections 2024-03-19T01:23:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- 172.21.15.7:0/424911026 wait complete. 2024-03-19T01:23:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 Processor -- start 2024-03-19T01:23:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- start start 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8340b3070 0x7ff8341417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff834142a60 0x7ff83413aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8341e0030 0x7ff8341e04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff834070e20 con 0x7ff8341e0030 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff834070ca0 con 0x7ff834142a60 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff8341524c0 con 0x7ff8340b3070 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff839239640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8341e0030 0x7ff8341e04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff838a38640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8340b3070 0x7ff8341417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff839239640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8341e0030 0x7ff8341e04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42776/0 (socket says 172.21.15.7:42776) 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff839239640 1 -- 172.21.15.7:0/3693826481 learned_addr learned my addr 172.21.15.7:0/3693826481 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff833fff640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff834142a60 0x7ff83413aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff839239640 1 -- 172.21.15.7:0/3693826481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8340b3070 msgr2=0x7ff8341417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff839239640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8340b3070 0x7ff8341417e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff839239640 1 -- 172.21.15.7:0/3693826481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff834142a60 msgr2=0x7ff83413aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff839239640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff834142a60 0x7ff83413aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff839239640 1 -- 172.21.15.7:0/3693826481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff834072190 con 0x7ff8341e0030 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff833fff640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff834142a60 0x7ff83413aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff839239640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8341e0030 0x7ff8341e04e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff82c007e40 tx=0x7ff82c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff8317fa640 1 -- 172.21.15.7:0/3693826481 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c014620 con 0x7ff8341e0030 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff8317fa640 1 -- 172.21.15.7:0/3693826481 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff82c019070 con 0x7ff8341e0030 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff8317fa640 1 -- 172.21.15.7:0/3693826481 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c014a20 con 0x7ff8341e0030 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff834072470 con 0x7ff8341e0030 2024-03-19T01:23:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.283+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff83413b5a0 con 0x7ff8341e0030 2024-03-19T01:23:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.287+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff83413b0c0 con 0x7ff8341e0030 2024-03-19T01:23:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.287+0000 7ff8317fa640 1 -- 172.21.15.7:0/3693826481 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff82c014bc0 con 0x7ff8341e0030 2024-03-19T01:23:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.287+0000 7ff8317fa640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff818067660 0x7ff818069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.287+0000 7ff8317fa640 1 -- 172.21.15.7:0/3693826481 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff82c0860d0 con 0x7ff8341e0030 2024-03-19T01:23:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.287+0000 7ff838a38640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff818067660 0x7ff818069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.287+0000 7ff838a38640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff818067660 0x7ff818069b20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff8200ae000 tx=0x7ff820009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:45.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.291+0000 7ff8317fa640 1 -- 172.21.15.7:0/3693826481 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff82c056d50 con 0x7ff8341e0030 2024-03-19T01:23:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.607+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff834075570 con 0x7ff8341e0030 2024-03-19T01:23:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.607+0000 7ff8317fa640 1 -- 172.21.15.7:0/3693826481 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7ff82c056650 con 0x7ff8341e0030 2024-03-19T01:23:45.608 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff818067660 msgr2=0x7ff818069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff818067660 0x7ff818069b20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff8200ae000 tx=0x7ff820009040 comp rx=0 tx=0).stop 2024-03-19T01:23:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8341e0030 msgr2=0x7ff8341e04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8341e0030 0x7ff8341e04e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff82c007e40 tx=0x7ff82c00e4b0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 shutdown_connections 2024-03-19T01:23:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff818067660 0x7ff818069b20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8341e0030 0x7ff8341e04e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff834142a60 0x7ff83413aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 --2- 172.21.15.7:0/3693826481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8340b3070 0x7ff8341417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 >> 172.21.15.7:0/3693826481 conn(0x7ff8340b9120 msgr2=0x7ff8340b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 shutdown_connections 2024-03-19T01:23:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.611+0000 7ff83ac9c640 1 -- 172.21.15.7:0/3693826481 wait complete. 2024-03-19T01:23:45.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f2643f83640 1 Processor -- start 2024-03-19T01:23:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f2643f83640 1 -- start start 2024-03-19T01:23:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f2643f83640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f263c072bb0 0x7f263c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f2643f83640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c14b490 0x7f263c141190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f2643f83640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f263c06d8e0 con 0x7f263c0b77b0 2024-03-19T01:23:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f2643f83640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f263c06dbc0 con 0x7f263c072bb0 2024-03-19T01:23:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f2643f83640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f263c0b7b90 con 0x7f263c14b490 2024-03-19T01:23:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f264151e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f263c072bb0 0x7f263c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f264151e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f263c072bb0 0x7f263c06d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42778/0 (socket says 172.21.15.7:42778) 2024-03-19T01:23:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f264151e640 1 -- 172.21.15.7:0/2845302784 learned_addr learned my addr 172.21.15.7:0/2845302784 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f2642520640 1 --2- 172.21.15.7:0/2845302784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c14b490 0x7f263c141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f264151e640 1 -- 172.21.15.7:0/2845302784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c14b490 msgr2=0x7f263c141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f264151e640 1 --2- 172.21.15.7:0/2845302784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c14b490 0x7f263c141190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f264151e640 1 -- 172.21.15.7:0/2845302784 >> v1:172.21.15.167:6789/0 conn(0x7f263c0b77b0 legacy=0x7f263c14ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.743+0000 7f264151e640 1 -- 172.21.15.7:0/2845302784 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263c06e270 con 0x7f263c072bb0 2024-03-19T01:23:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f264151e640 1 --2- 172.21.15.7:0/2845302784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 0x7f263c06d2a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f262c009870 tx=0x7f262c01c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66041a2a1694bf07 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2640d1d640 1 -- 172.21.15.7:0/2845302784 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c0b0070 con 0x7f263c072bb0 2024-03-19T01:23:45.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2640d1d640 1 -- 172.21.15.7:0/2845302784 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262c0ab600 con 0x7f263c072bb0 2024-03-19T01:23:45.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2640d1d640 1 -- 172.21.15.7:0/2845302784 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c0ab9d0 con 0x7f263c072bb0 2024-03-19T01:23:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2643f83640 1 -- 172.21.15.7:0/2845302784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 msgr2=0x7f263c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2643f83640 1 --2- 172.21.15.7:0/2845302784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 0x7f263c06d2a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f262c009870 tx=0x7f262c01c2a0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2643f83640 1 -- 172.21.15.7:0/2845302784 shutdown_connections 2024-03-19T01:23:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2643f83640 1 --2- 172.21.15.7:0/2845302784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c14b490 0x7f263c141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2643f83640 1 --2- 172.21.15.7:0/2845302784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 0x7f263c06d2a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2643f83640 1 -- 172.21.15.7:0/2845302784 >> 172.21.15.7:0/2845302784 conn(0x7f263c06b2c0 msgr2=0x7f263c06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2643f83640 1 -- 172.21.15.7:0/2845302784 shutdown_connections 2024-03-19T01:23:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.747+0000 7f2643f83640 1 -- 172.21.15.7:0/2845302784 wait complete. 2024-03-19T01:23:45.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 Processor -- start 2024-03-19T01:23:45.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 -- start start 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 0x7f263c147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c0b77b0 0x7f263c1e3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f263c14b490 0x7f263c1e39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f263c154880 con 0x7f263c072bb0 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f263c154700 con 0x7f263c14b490 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f263c154a00 con 0x7f263c0b77b0 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2641d1f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 0x7f263c147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2641d1f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 0x7f263c147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42794/0 (socket says 172.21.15.7:42794) 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2641d1f640 1 -- 172.21.15.7:0/1197238845 learned_addr learned my addr 172.21.15.7:0/1197238845 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2642520640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f263c14b490 0x7f263c1e39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f264151e640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c0b77b0 0x7f263c1e3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2641d1f640 1 -- 172.21.15.7:0/1197238845 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c0b77b0 msgr2=0x7f263c1e3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2641d1f640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c0b77b0 0x7f263c1e3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2641d1f640 1 -- 172.21.15.7:0/1197238845 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f263c14b490 msgr2=0x7f263c1e39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2641d1f640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f263c14b490 0x7f263c1e39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2641d1f640 1 -- 172.21.15.7:0/1197238845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263c146600 con 0x7f263c072bb0 2024-03-19T01:23:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2641d1f640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 0x7f263c147800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f263800c450 tx=0x7f263800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f26327fc640 1 -- 172.21.15.7:0/1197238845 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2638016020 con 0x7f263c072bb0 2024-03-19T01:23:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f26327fc640 1 -- 172.21.15.7:0/1197238845 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26380052d0 con 0x7f263c072bb0 2024-03-19T01:23:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f26327fc640 1 -- 172.21.15.7:0/1197238845 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2638007260 con 0x7f263c072bb0 2024-03-19T01:23:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263c1346f0 con 0x7f263c072bb0 2024-03-19T01:23:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f263c144e40 con 0x7f263c072bb0 2024-03-19T01:23:45.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f263c1450d0 con 0x7f263c072bb0 2024-03-19T01:23:45.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.751+0000 7f26327fc640 1 -- 172.21.15.7:0/1197238845 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2638007400 con 0x7f263c072bb0 2024-03-19T01:23:45.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.755+0000 7f26327fc640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2620067660 0x7f2620069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:45.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.755+0000 7f26327fc640 1 -- 172.21.15.7:0/1197238845 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f26380851f0 con 0x7f263c072bb0 2024-03-19T01:23:45.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.755+0000 7f264151e640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2620067660 0x7f2620069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:45.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.755+0000 7f264151e640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2620067660 0x7f2620069b20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f262c0ad000 tx=0x7f262c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:45.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:45.759+0000 7f26327fc640 1 -- 172.21.15.7:0/1197238845 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2638020020 con 0x7f263c072bb0 2024-03-19T01:23:46.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f263c075570 con 0x7f263c072bb0 2024-03-19T01:23:46.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f26327fc640 1 -- 172.21.15.7:0/1197238845 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f2638005480 con 0x7f263c072bb0 2024-03-19T01:23:46.104 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:46.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2620067660 msgr2=0x7f2620069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2620067660 0x7f2620069b20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f262c0ad000 tx=0x7f262c008040 comp rx=0 tx=0).stop 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 msgr2=0x7f263c147800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 0x7f263c147800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f263800c450 tx=0x7f263800c910 comp rx=0 tx=0).stop 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 shutdown_connections 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2620067660 0x7f2620069b20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f263c0b77b0 0x7f263c1e3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f263c14b490 0x7f263c1e39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 --2- 172.21.15.7:0/1197238845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f263c072bb0 0x7f263c147800 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 >> 172.21.15.7:0/1197238845 conn(0x7f263c06b2c0 msgr2=0x7f263c1487a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 shutdown_connections 2024-03-19T01:23:46.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.103+0000 7f2643f83640 1 -- 172.21.15.7:0/1197238845 wait complete. 2024-03-19T01:23:46.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:46.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.251+0000 7f3b7a10e640 1 Processor -- start 2024-03-19T01:23:46.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.251+0000 7f3b7a10e640 1 -- start start 2024-03-19T01:23:46.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b7a10e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3b740b5680 0x7f3b7406e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b7a10e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f3b740b0470 0x7f3b7406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b7a10e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b740af2f0 con 0x7f3b74072c50 2024-03-19T01:23:46.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b7a10e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f3b740b5a60 con 0x7f3b740b5680 2024-03-19T01:23:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b7a10e640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f3b740afb50 con 0x7f3b740b0470 2024-03-19T01:23:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b737fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3b740b5680 0x7f3b7406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b737fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3b740b5680 0x7f3b7406e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42796/0 (socket says 172.21.15.7:42796) 2024-03-19T01:23:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b737fe640 1 -- 172.21.15.7:0/2305938877 learned_addr learned my addr 172.21.15.7:0/2305938877 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b73fff640 1 --2- 172.21.15.7:0/2305938877 >> v2:172.21.15.167:3300/0 conn(0x7f3b740b0470 0x7f3b7406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b737fe640 1 -- 172.21.15.7:0/2305938877 >> v2:172.21.15.167:3300/0 conn(0x7f3b740b0470 msgr2=0x7f3b7406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b737fe640 1 --2- 172.21.15.7:0/2305938877 >> v2:172.21.15.167:3300/0 conn(0x7f3b740b0470 0x7f3b7406d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b737fe640 1 -- 172.21.15.7:0/2305938877 >> v1:172.21.15.167:6789/0 conn(0x7f3b74072c50 legacy=0x7f3b7406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b737fe640 1 -- 172.21.15.7:0/2305938877 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b740b09c0 con 0x7f3b740b5680 2024-03-19T01:23:46.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b737fe640 1 --2- 172.21.15.7:0/2305938877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b5680 0x7f3b7406e860 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3b74073030 tx=0x7f3b6c01c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54cac95a47d2adf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b727fc640 1 -- 172.21.15.7:0/2305938877 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c0b2070 con 0x7f3b740b5680 2024-03-19T01:23:46.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b727fc640 1 -- 172.21.15.7:0/2305938877 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b6c0ad640 con 0x7f3b740b5680 2024-03-19T01:23:46.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.255+0000 7f3b727fc640 1 -- 172.21.15.7:0/2305938877 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c0ada10 con 0x7f3b740b5680 2024-03-19T01:23:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- 172.21.15.7:0/2305938877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b5680 msgr2=0x7f3b7406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 --2- 172.21.15.7:0/2305938877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b5680 0x7f3b7406e860 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3b74073030 tx=0x7f3b6c01c390 comp rx=0 tx=0).stop 2024-03-19T01:23:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- 172.21.15.7:0/2305938877 shutdown_connections 2024-03-19T01:23:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 --2- 172.21.15.7:0/2305938877 >> v2:172.21.15.167:3300/0 conn(0x7f3b740b0470 0x7f3b7406d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 --2- 172.21.15.7:0/2305938877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b5680 0x7f3b7406e860 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- 172.21.15.7:0/2305938877 >> 172.21.15.7:0/2305938877 conn(0x7f3b740b6510 msgr2=0x7f3b740b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- 172.21.15.7:0/2305938877 shutdown_connections 2024-03-19T01:23:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- 172.21.15.7:0/2305938877 wait complete. 2024-03-19T01:23:46.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 Processor -- start 2024-03-19T01:23:46.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- start start 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b74072c50 0x7f3b7409c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b0470 0x7f3b741db600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b740b5680 0x7f3b7409d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3b74147a40 con 0x7f3b740b0470 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3b741478c0 con 0x7f3b74072c50 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3b7406dc30 con 0x7f3b740b5680 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b72ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b0470 0x7f3b741db600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b72ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b0470 0x7f3b741db600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42800/0 (socket says 172.21.15.7:42800) 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b72ffd640 1 -- 172.21.15.7:0/1827126614 learned_addr learned my addr 172.21.15.7:0/1827126614 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b73fff640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b740b5680 0x7f3b7409d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b737fe640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b74072c50 0x7f3b7409c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b72ffd640 1 -- 172.21.15.7:0/1827126614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b740b5680 msgr2=0x7f3b7409d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b72ffd640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b740b5680 0x7f3b7409d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b72ffd640 1 -- 172.21.15.7:0/1827126614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b74072c50 msgr2=0x7f3b7409c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b72ffd640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b74072c50 0x7f3b7409c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b72ffd640 1 -- 172.21.15.7:0/1827126614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7409b150 con 0x7f3b740b0470 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b737fe640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b74072c50 0x7f3b7409c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b72ffd640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b0470 0x7f3b741db600 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00c7b0 tx=0x7f3b5c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b6bfff640 1 -- 172.21.15.7:0/1827126614 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c016030 con 0x7f3b740b0470 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b6bfff640 1 -- 172.21.15.7:0/1827126614 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b5c006b60 con 0x7f3b740b0470 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b6bfff640 1 -- 172.21.15.7:0/1827126614 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c00a850 con 0x7f3b740b0470 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.259+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b7409dc60 con 0x7f3b740b0470 2024-03-19T01:23:46.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.263+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7409b8e0 con 0x7f3b740b0470 2024-03-19T01:23:46.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.263+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b7409b5e0 con 0x7f3b740b0470 2024-03-19T01:23:46.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.263+0000 7f3b6bfff640 1 -- 172.21.15.7:0/1827126614 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3b5c019070 con 0x7f3b740b0470 2024-03-19T01:23:46.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.263+0000 7f3b6bfff640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f3b58067660 0x7f3b58069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.263+0000 7f3b737fe640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f3b58067660 0x7f3b58069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.263+0000 7f3b6bfff640 1 -- 172.21.15.7:0/1827126614 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3b5c085760 con 0x7f3b740b0470 2024-03-19T01:23:46.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.263+0000 7f3b737fe640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f3b58067660 0x7f3b58069b20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0af000 tx=0x7f3b6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.267+0000 7f3b6bfff640 1 -- 172.21.15.7:0/1827126614 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3b7409b5e0 con 0x7f3b740b0470 2024-03-19T01:23:46.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.443+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b7409cbd0 con 0x7f3b740b0470 2024-03-19T01:23:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.443+0000 7f3b6bfff640 1 -- 172.21.15.7:0/1827126614 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7f3b5c056470 con 0x7f3b740b0470 2024-03-19T01:23:46.445 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:46.445 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.443+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f3b58067660 msgr2=0x7f3b58069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.443+0000 7f3b7a10e640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f3b58067660 0x7f3b58069b20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0af000 tx=0x7f3b6c008040 comp rx=0 tx=0).stop 2024-03-19T01:23:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.443+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b0470 msgr2=0x7f3b741db600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.443+0000 7f3b7a10e640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b0470 0x7f3b741db600 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00c7b0 tx=0x7f3b5c00cc70 comp rx=0 tx=0).stop 2024-03-19T01:23:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.447+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 shutdown_connections 2024-03-19T01:23:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.447+0000 7f3b7a10e640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f3b58067660 0x7f3b58069b20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.447+0000 7f3b7a10e640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b740b5680 0x7f3b7409d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.447+0000 7f3b7a10e640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b740b0470 0x7f3b741db600 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.447+0000 7f3b7a10e640 1 --2- 172.21.15.7:0/1827126614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b74072c50 0x7f3b7409c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.447+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 >> 172.21.15.7:0/1827126614 conn(0x7f3b740b6510 msgr2=0x7f3b741426d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.447+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 shutdown_connections 2024-03-19T01:23:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.447+0000 7f3b7a10e640 1 -- 172.21.15.7:0/1827126614 wait complete. 2024-03-19T01:23:46.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:46.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:46.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.595+0000 7fa4dfbc3640 1 Processor -- start 2024-03-19T01:23:46.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4dfbc3640 1 -- start start 2024-03-19T01:23:46.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4dfbc3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b56e0 0x7fa4d80b4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4dfbc3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4d80b48c0 0x7fa4d814fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4dfbc3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4d80b2c20 con 0x7fa4d80b77f0 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4dfbc3640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d80b5d00 con 0x7fa4d80b48c0 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4dfbc3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d80b5fe0 con 0x7fa4d80b56e0 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4de160640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4d80b48c0 0x7fa4d814fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4de160640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4d80b48c0 0x7fa4d814fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42808/0 (socket says 172.21.15.7:42808) 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4de160640 1 -- 172.21.15.7:0/2846300278 learned_addr learned my addr 172.21.15.7:0/2846300278 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4dd15e640 1 --2- 172.21.15.7:0/2846300278 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b56e0 0x7fa4d80b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4de160640 1 -- 172.21.15.7:0/2846300278 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b56e0 msgr2=0x7fa4d80b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4de160640 1 --2- 172.21.15.7:0/2846300278 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b56e0 0x7fa4d80b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4de160640 1 -- 172.21.15.7:0/2846300278 >> v1:172.21.15.167:6789/0 conn(0x7fa4d80b77f0 legacy=0x7fa4d80b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4de160640 1 -- 172.21.15.7:0/2846300278 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d80b64d0 con 0x7fa4d80b48c0 2024-03-19T01:23:46.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.599+0000 7fa4dd15e640 1 --2- 172.21.15.7:0/2846300278 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b56e0 0x7fa4d80b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:46.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4de160640 1 --2- 172.21.15.7:0/2846300278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 0x7fa4d814fb60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009870 tx=0x7fa4d401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ce02b9e4f2b034 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dc95d640 1 -- 172.21.15.7:0/2846300278 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40b1070 con 0x7fa4d80b48c0 2024-03-19T01:23:46.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dc95d640 1 -- 172.21.15.7:0/2846300278 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4d40a8050 con 0x7fa4d80b48c0 2024-03-19T01:23:46.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dc95d640 1 -- 172.21.15.7:0/2846300278 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40ac860 con 0x7fa4d80b48c0 2024-03-19T01:23:46.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2846300278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 msgr2=0x7fa4d814fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 --2- 172.21.15.7:0/2846300278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 0x7fa4d814fb60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009870 tx=0x7fa4d401c4d0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2846300278 shutdown_connections 2024-03-19T01:23:46.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 --2- 172.21.15.7:0/2846300278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 0x7fa4d814fb60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 --2- 172.21.15.7:0/2846300278 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b56e0 0x7fa4d80b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2846300278 >> 172.21.15.7:0/2846300278 conn(0x7fa4d806b2c0 msgr2=0x7fa4d806b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2846300278 shutdown_connections 2024-03-19T01:23:46.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2846300278 wait complete. 2024-03-19T01:23:46.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 Processor -- start 2024-03-19T01:23:46.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 -- start start 2024-03-19T01:23:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 0x7fa4d81dd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4d80b56e0 0x7fa4d81421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b77f0 0x7fa4d81412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d81489b0 con 0x7fa4d80b48c0 2024-03-19T01:23:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d8148830 con 0x7fa4d80b56e0 2024-03-19T01:23:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dfbc3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d81528c0 con 0x7fa4d80b77f0 2024-03-19T01:23:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dd95f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 0x7fa4d81dd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dd95f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 0x7fa4d81dd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42824/0 (socket says 172.21.15.7:42824) 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dd95f640 1 -- 172.21.15.7:0/2813942536 learned_addr learned my addr 172.21.15.7:0/2813942536 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4de160640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b77f0 0x7fa4d81412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dd15e640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4d80b56e0 0x7fa4d81421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dd95f640 1 -- 172.21.15.7:0/2813942536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b77f0 msgr2=0x7fa4d81412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dd95f640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b77f0 0x7fa4d81412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dd95f640 1 -- 172.21.15.7:0/2813942536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4d80b56e0 msgr2=0x7fa4d81421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dd95f640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4d80b56e0 0x7fa4d81421a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.603+0000 7fa4dd95f640 1 -- 172.21.15.7:0/2813942536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d8141da0 con 0x7fa4d80b48c0 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4dd95f640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 0x7fa4d81dd7b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000c7a0 tx=0x7fa4d000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4ce7fc640 1 -- 172.21.15.7:0/2813942536 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0003b80 con 0x7fa4d80b48c0 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4ce7fc640 1 -- 172.21.15.7:0/2813942536 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4d0003d20 con 0x7fa4d80b48c0 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d80b8c10 con 0x7fa4d80b48c0 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4ce7fc640 1 -- 172.21.15.7:0/2813942536 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d000aa90 con 0x7fa4d80b48c0 2024-03-19T01:23:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d81ddcf0 con 0x7fa4d80b48c0 2024-03-19T01:23:46.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d8075570 con 0x7fa4d80b48c0 2024-03-19T01:23:46.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4ce7fc640 1 -- 172.21.15.7:0/2813942536 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa4d000ac30 con 0x7fa4d80b48c0 2024-03-19T01:23:46.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4ce7fc640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa4bc067660 0x7fa4bc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4ce7fc640 1 -- 172.21.15.7:0/2813942536 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa4d0085730 con 0x7fa4d80b48c0 2024-03-19T01:23:46.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4dd15e640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa4bc067660 0x7fa4bc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.607+0000 7fa4dd15e640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa4bc067660 0x7fa4bc069b20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa4d813f090 tx=0x7fa4c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.611+0000 7fa4ce7fc640 1 -- 172.21.15.7:0/2813942536 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa4d0056440 con 0x7fa4d80b48c0 2024-03-19T01:23:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.771+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa4d80b23b0 con 0x7fa4d80b48c0 2024-03-19T01:23:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.771+0000 7fa4ce7fc640 1 -- 172.21.15.7:0/2813942536 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fa4d0055d40 con 0x7fa4d80b48c0 2024-03-19T01:23:46.774 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:46.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa4bc067660 msgr2=0x7fa4bc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa4bc067660 0x7fa4bc069b20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa4d813f090 tx=0x7fa4c8008040 comp rx=0 tx=0).stop 2024-03-19T01:23:46.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 msgr2=0x7fa4d81dd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 0x7fa4d81dd7b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000c7a0 tx=0x7fa4d000cc60 comp rx=0 tx=0).stop 2024-03-19T01:23:46.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 shutdown_connections 2024-03-19T01:23:46.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa4bc067660 0x7fa4bc069b20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4d80b77f0 0x7fa4d81412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4d80b56e0 0x7fa4d81421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 --2- 172.21.15.7:0/2813942536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4d80b48c0 0x7fa4d81dd7b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 >> 172.21.15.7:0/2813942536 conn(0x7fa4d806b2c0 msgr2=0x7fa4d8144620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 shutdown_connections 2024-03-19T01:23:46.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.775+0000 7fa4dfbc3640 1 -- 172.21.15.7:0/2813942536 wait complete. 2024-03-19T01:23:46.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:46.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f7b738640 1 Processor -- start 2024-03-19T01:23:46.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f7b738640 1 -- start start 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f7b738640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b5780 0x7f1f7414aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f7b738640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f740afee0 con 0x7f1f7414fd80 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f7b738640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f740b5b60 con 0x7f1f74149e50 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f7b738640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1f740b6250 con 0x7f1f740b5780 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f79cd5640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1f7414fd80 0x7f1f74152170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35048/0 (socket says 172.21.15.7:35048) 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f79cd5640 1 -- 172.21.15.7:0/2118840161 learned_addr learned my addr 172.21.15.7:0/2118840161 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456115228 0 0) 0x7f1f740afee0 con 0x7f1f7414fd80 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f794d4640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b5780 0x7f1f7414aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f60003530 con 0x7f1f7414fd80 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2409670950 0 0) 0x7f1f60003530 con 0x7f1f7414fd80 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b5780 msgr2=0x7f1f7414aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b5780 0x7f1f7414aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 >> v1:172.21.15.167:6789/0 conn(0x7f1f74149e50 legacy=0x7f1f7414a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f740aec10 con 0x7f1f7414fd80 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f1f6c001680 con 0x7f1f7414fd80 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 >> v1:172.21.15.7:6789/0 conn(0x7f1f7414fd80 legacy=0x7f1f74152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f60003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f60004810 0x7f1f60004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b5780 0x7f1f60005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1f740b6250 con 0x7f1f600039f0 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6000bb90 con 0x7f1f60004810 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6000bd90 con 0x7f1f740b5780 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f794d4640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f60003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f794d4640 1 -- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b5780 msgr2=0x7f1f60005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f794d4640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b5780 0x7f1f60005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f794d4640 1 -- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f60004810 msgr2=0x7f1f60004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f78cd3640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f60004810 0x7f1f60004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f794d4640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f60004810 0x7f1f60004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.903+0000 7f1f794d4640 1 -- 172.21.15.7:0/2118840161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f60003530 con 0x7f1f600039f0 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f794d4640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f60003e30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009fc0 tx=0x7f1f7001ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa2a015aaf15b96 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f700b2070 con 0x7f1f600039f0 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f700a59d0 con 0x7f1f600039f0 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f6bfff640 1 -- 172.21.15.7:0/2118840161 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f70009c10 con 0x7f1f600039f0 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f7b738640 1 -- 172.21.15.7:0/2118840161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 msgr2=0x7f1f60003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f7b738640 1 --2- 172.21.15.7:0/2118840161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f60003e30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009fc0 tx=0x7f1f7001ccc0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f79cd5640 1 -- 172.21.15.7:0/2118840161 reap_dead start 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f7b738640 1 -- 172.21.15.7:0/2118840161 shutdown_connections 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f7b738640 1 -- 172.21.15.7:0/2118840161 >> 172.21.15.7:0/2118840161 conn(0x7f1f740b6610 msgr2=0x7f1f740b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f7b738640 1 -- 172.21.15.7:0/2118840161 shutdown_connections 2024-03-19T01:23:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f7b738640 1 -- 172.21.15.7:0/2118840161 wait complete. 2024-03-19T01:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f7b738640 1 Processor -- start 2024-03-19T01:23:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.907+0000 7f1f7b738640 1 -- start start 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f7b738640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f740b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f7b738640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f740b2f40 0x7f1f741df7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f7b738640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b39a0 0x7f1f74152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f7b738640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1f74072d70 con 0x7f1f600039f0 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f7b738640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1f740afa50 con 0x7f1f740b2f40 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f7b738640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1f74072ef0 con 0x7f1f740b39a0 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f794d4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f740b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f794d4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f740b2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42838/0 (socket says 172.21.15.7:42838) 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f794d4640 1 -- 172.21.15.7:0/2144367967 learned_addr learned my addr 172.21.15.7:0/2144367967 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f78cd3640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f740b2f40 0x7f1f741df7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f794d4640 1 -- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b39a0 msgr2=0x7f1f74152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f79cd5640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b39a0 0x7f1f74152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f794d4640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b39a0 0x7f1f74152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f794d4640 1 -- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f740b2f40 msgr2=0x7f1f741df7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f794d4640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f740b2f40 0x7f1f741df7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f794d4640 1 -- 172.21.15.7:0/2144367967 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f740b18d0 con 0x7f1f600039f0 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f78cd3640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f740b2f40 0x7f1f741df7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f79cd5640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b39a0 0x7f1f74152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f794d4640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f740b2a00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1f700984f0 tx=0x7f1f70009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f69ffb640 1 -- 172.21.15.7:0/2144367967 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f700ba070 con 0x7f1f600039f0 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f69ffb640 1 -- 172.21.15.7:0/2144367967 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f700039a0 con 0x7f1f600039f0 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f69ffb640 1 -- 172.21.15.7:0/2144367967 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f700b2040 con 0x7f1f600039f0 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f740b2060 con 0x7f1f600039f0 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f740b2300 con 0x7f1f600039f0 2024-03-19T01:23:46.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f74075570 con 0x7f1f600039f0 2024-03-19T01:23:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f69ffb640 1 -- 172.21.15.7:0/2144367967 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1f700ad440 con 0x7f1f600039f0 2024-03-19T01:23:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f69ffb640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f58067690 0x7f1f58069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f69ffb640 1 -- 172.21.15.7:0/2144367967 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1f7011fc90 con 0x7f1f600039f0 2024-03-19T01:23:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f78cd3640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f58067690 0x7f1f58069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.911+0000 7f1f78cd3640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f58067690 0x7f1f58069b50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1f740af7b0 tx=0x7f1f64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:46.915+0000 7f1f69ffb640 1 -- 172.21.15.7:0/2144367967 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1f700169d0 con 0x7f1f600039f0 2024-03-19T01:23:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1f740b0be0 con 0x7f1f600039f0 2024-03-19T01:23:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f69ffb640 1 -- 172.21.15.7:0/2144367967 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f1f700f0350 con 0x7f1f600039f0 2024-03-19T01:23:47.072 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f58067690 msgr2=0x7f1f58069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f58067690 0x7f1f58069b50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1f740af7b0 tx=0x7f1f64009040 comp rx=0 tx=0).stop 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 msgr2=0x7f1f740b2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f740b2a00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1f700984f0 tx=0x7f1f70009750 comp rx=0 tx=0).stop 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 shutdown_connections 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1f58067690 0x7f1f58069b50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f740b39a0 0x7f1f74152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f740b2f40 0x7f1f741df7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 --2- 172.21.15.7:0/2144367967 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f600039f0 0x7f1f740b2a00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 >> 172.21.15.7:0/2144367967 conn(0x7f1f740b6610 msgr2=0x7f1f7414f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 shutdown_connections 2024-03-19T01:23:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.071+0000 7f1f7b738640 1 -- 172.21.15.7:0/2144367967 wait complete. 2024-03-19T01:23:47.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:47.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a17887640 1 Processor -- start 2024-03-19T01:23:47.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a17887640 1 -- start start 2024-03-19T01:23:47.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a17887640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f1a10149e50 0x7f1a1014a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a17887640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1014fd80 0x7f1a10152170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a17887640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1014a820 con 0x7f1a100b5780 2024-03-19T01:23:47.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a17887640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f1a100b6250 con 0x7f1a10149e50 2024-03-19T01:23:47.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a17887640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a100afa50 con 0x7f1a1014fd80 2024-03-19T01:23:47.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a15623640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1a100b5780 0x7f1a1014aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35052/0 (socket says 172.21.15.7:35052) 2024-03-19T01:23:47.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a15623640 1 -- 172.21.15.7:0/1756174389 learned_addr learned my addr 172.21.15.7:0/1756174389 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a15e24640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1014fd80 0x7f1a10152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a14e22640 1 --2- 172.21.15.7:0/1756174389 >> v2:172.21.15.167:3300/0 conn(0x7f1a10149e50 0x7f1a1014a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68788193 0 0) 0x7f1a1014a820 con 0x7f1a100b5780 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19fc003530 con 0x7f1a100b5780 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 708487894 0 0) 0x7f19fc003530 con 0x7f1a100b5780 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1014fd80 msgr2=0x7f1a10152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1014fd80 0x7f1a10152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 >> v2:172.21.15.167:3300/0 conn(0x7f1a10149e50 msgr2=0x7f1a1014a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 --2- 172.21.15.7:0/1756174389 >> v2:172.21.15.167:3300/0 conn(0x7f1a10149e50 0x7f1a1014a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a100afee0 con 0x7f1a100b5780 2024-03-19T01:23:47.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f1a0c001680 con 0x7f1a100b5780 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 >> v1:172.21.15.7:6789/0 conn(0x7f1a100b5780 legacy=0x7f1a1014aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1014fd80 0x7f19fc0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f19fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc004d50 0x7f19fc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1a100b6250 con 0x7f19fc003f40 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a100afa50 con 0x7f19fc004d50 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f19fc00ba20 con 0x7f1a1014fd80 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a14e22640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f19fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a15623640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1014fd80 0x7f19fc0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a15e24640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc004d50 0x7f19fc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a14e22640 1 -- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1014fd80 msgr2=0x7f19fc0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a14e22640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1014fd80 0x7f19fc0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a14e22640 1 -- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc004d50 msgr2=0x7f19fc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a14e22640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc004d50 0x7f19fc0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.203+0000 7f1a14e22640 1 -- 172.21.15.7:0/1756174389 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19fc003530 con 0x7f19fc003f40 2024-03-19T01:23:47.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a14e22640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f19fc004810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009b10 tx=0x7f1a0001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5042f6cb7b965f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a000096b0 con 0x7f19fc003f40 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a000a6490 con 0x7f19fc003f40 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a07fff640 1 -- 172.21.15.7:0/1756174389 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a000a6880 con 0x7f19fc003f40 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- 172.21.15.7:0/1756174389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 msgr2=0x7f19fc004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 --2- 172.21.15.7:0/1756174389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f19fc004810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009b10 tx=0x7f1a0001cab0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15e24640 1 -- 172.21.15.7:0/1756174389 reap_dead start 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- 172.21.15.7:0/1756174389 shutdown_connections 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- 172.21.15.7:0/1756174389 >> 172.21.15.7:0/1756174389 conn(0x7f1a100b6610 msgr2=0x7f1a100b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- 172.21.15.7:0/1756174389 shutdown_connections 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- 172.21.15.7:0/1756174389 wait complete. 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 Processor -- start 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- start start 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f1a10148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a10147b40 0x7f1a10146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a10146ed0 0x7f1a101401e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1006d2a0 con 0x7f19fc003f40 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1014b170 con 0x7f1a10146ed0 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1006ec60 con 0x7f1a10147b40 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15623640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f1a10148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15623640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f1a10148cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42854/0 (socket says 172.21.15.7:42854) 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15623640 1 -- 172.21.15.7:0/1068112590 learned_addr learned my addr 172.21.15.7:0/1068112590 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a14e22640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a10147b40 0x7f1a10146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15e24640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a10146ed0 0x7f1a101401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15623640 1 -- 172.21.15.7:0/1068112590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a10147b40 msgr2=0x7f1a10146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15623640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a10147b40 0x7f1a10146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15623640 1 -- 172.21.15.7:0/1068112590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a10146ed0 msgr2=0x7f1a101401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15623640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a10146ed0 0x7f1a101401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15623640 1 -- 172.21.15.7:0/1068112590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a10147350 con 0x7f19fc003f40 2024-03-19T01:23:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a14e22640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a10147b40 0x7f1a10146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a15623640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f1a10148cb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00e7e0 tx=0x7f1a0c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a05ffb640 1 -- 172.21.15.7:0/1068112590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c00ee90 con 0x7f19fc003f40 2024-03-19T01:23:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a05ffb640 1 -- 172.21.15.7:0/1068112590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c018070 con 0x7f19fc003f40 2024-03-19T01:23:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.207+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a10147fc0 con 0x7f19fc003f40 2024-03-19T01:23:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.211+0000 7f1a05ffb640 1 -- 172.21.15.7:0/1068112590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c00ca10 con 0x7f19fc003f40 2024-03-19T01:23:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.211+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a100b3630 con 0x7f19fc003f40 2024-03-19T01:23:47.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.211+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a10075570 con 0x7f19fc003f40 2024-03-19T01:23:47.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.211+0000 7f1a05ffb640 1 -- 172.21.15.7:0/1068112590 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1a0c00cbb0 con 0x7f19fc003f40 2024-03-19T01:23:47.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.211+0000 7f1a05ffb640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f19f4067800 0x7f19f4069cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.211+0000 7f1a14e22640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f19f4067800 0x7f19f4069cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.211+0000 7f1a05ffb640 1 -- 172.21.15.7:0/1068112590 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1a0c086180 con 0x7f19fc003f40 2024-03-19T01:23:47.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.211+0000 7f1a14e22640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f19f4067800 0x7f19f4069cc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1a000ad000 tx=0x7f1a000b2040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:47.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.215+0000 7f1a05ffb640 1 -- 172.21.15.7:0/1068112590 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1a10147350 con 0x7f19fc003f40 2024-03-19T01:23:47.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.367+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1a100b8170 con 0x7f19fc003f40 2024-03-19T01:23:47.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a05ffb640 1 -- 172.21.15.7:0/1068112590 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f1a0c056e70 con 0x7f19fc003f40 2024-03-19T01:23:47.371 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:47.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f19f4067800 msgr2=0x7f19f4069cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f19f4067800 0x7f19f4069cc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1a000ad000 tx=0x7f1a000b2040 comp rx=0 tx=0).stop 2024-03-19T01:23:47.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 msgr2=0x7f1a10148cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f1a10148cb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00e7e0 tx=0x7f1a0c00eca0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 shutdown_connections 2024-03-19T01:23:47.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f19f4067800 0x7f19f4069cc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a10147b40 0x7f1a10146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a10146ed0 0x7f1a101401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 --2- 172.21.15.7:0/1068112590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19fc003f40 0x7f1a10148cb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 >> 172.21.15.7:0/1068112590 conn(0x7f1a100b6610 msgr2=0x7f1a1006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:47.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 shutdown_connections 2024-03-19T01:23:47.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.371+0000 7f1a17887640 1 -- 172.21.15.7:0/1068112590 wait complete. 2024-03-19T01:23:47.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:47.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.499+0000 7f22e4572640 1 Processor -- start 2024-03-19T01:23:47.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.499+0000 7f22e4572640 1 -- start start 2024-03-19T01:23:47.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e4572640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc13f460 0x7f22dc074130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e4572640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22dc06fe80 0x7f22dc148990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e4572640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22dc071160 con 0x7f22dc148ed0 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e4572640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f22dc0b3550 con 0x7f22dc06fe80 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e4572640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f22dc149340 con 0x7f22dc13f460 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e1b0d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22dc06fe80 0x7f22dc148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e1b0d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22dc06fe80 0x7f22dc148990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42858/0 (socket says 172.21.15.7:42858) 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e1b0d640 1 -- 172.21.15.7:0/4294391159 learned_addr learned my addr 172.21.15.7:0/4294391159 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e230e640 1 --2- 172.21.15.7:0/4294391159 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc13f460 0x7f22dc074130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e1b0d640 1 -- 172.21.15.7:0/4294391159 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc13f460 msgr2=0x7f22dc074130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e1b0d640 1 --2- 172.21.15.7:0/4294391159 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc13f460 0x7f22dc074130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e1b0d640 1 -- 172.21.15.7:0/4294391159 >> v1:172.21.15.167:6789/0 conn(0x7f22dc148ed0 legacy=0x7f22dc0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e1b0d640 1 -- 172.21.15.7:0/4294391159 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc0b2f00 con 0x7f22dc06fe80 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e1b0d640 1 --2- 172.21.15.7:0/4294391159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc06fe80 0x7f22dc148990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009870 tx=0x7f22d801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee596ed0a9aa181 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e130c640 1 -- 172.21.15.7:0/4294391159 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d80b0070 con 0x7f22dc06fe80 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e130c640 1 -- 172.21.15.7:0/4294391159 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22d80ab400 con 0x7f22dc06fe80 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.503+0000 7f22e130c640 1 -- 172.21.15.7:0/4294391159 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d80ab7d0 con 0x7f22dc06fe80 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- 172.21.15.7:0/4294391159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc06fe80 msgr2=0x7f22dc148990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 --2- 172.21.15.7:0/4294391159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc06fe80 0x7f22dc148990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009870 tx=0x7f22d801c490 comp rx=0 tx=0).stop 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- 172.21.15.7:0/4294391159 shutdown_connections 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 --2- 172.21.15.7:0/4294391159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc06fe80 0x7f22dc148990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 --2- 172.21.15.7:0/4294391159 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc13f460 0x7f22dc074130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- 172.21.15.7:0/4294391159 >> 172.21.15.7:0/4294391159 conn(0x7f22dc0b5c20 msgr2=0x7f22dc0b6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- 172.21.15.7:0/4294391159 shutdown_connections 2024-03-19T01:23:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- 172.21.15.7:0/4294391159 wait complete. 2024-03-19T01:23:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 Processor -- start 2024-03-19T01:23:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- start start 2024-03-19T01:23:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc06fe80 0x7f22dc1446d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc13f460 0x7f22dc1dde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22dc148ed0 0x7f22dc1de3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc0b5030 con 0x7f22dc13f460 2024-03-19T01:23:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc0b4eb0 con 0x7f22dc148ed0 2024-03-19T01:23:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f22dc152bd0 con 0x7f22dc06fe80 2024-03-19T01:23:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e1b0d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc13f460 0x7f22dc1dde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e1b0d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc13f460 0x7f22dc1dde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42864/0 (socket says 172.21.15.7:42864) 2024-03-19T01:23:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e1b0d640 1 -- 172.21.15.7:0/2803058608 learned_addr learned my addr 172.21.15.7:0/2803058608 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e230e640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc06fe80 0x7f22dc1446d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e1b0d640 1 -- 172.21.15.7:0/2803058608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc06fe80 msgr2=0x7f22dc1446d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e1b0d640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc06fe80 0x7f22dc1446d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e1b0d640 1 -- 172.21.15.7:0/2803058608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22dc148ed0 msgr2=0x7f22dc1de3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e1b0d640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22dc148ed0 0x7f22dc1de3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e1b0d640 1 -- 172.21.15.7:0/2803058608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc134480 con 0x7f22dc13f460 2024-03-19T01:23:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e1b0d640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc13f460 0x7f22dc1dde60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f22d801cd30 tx=0x7f22d8019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22d2ffd640 1 -- 172.21.15.7:0/2803058608 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d80b0040 con 0x7f22dc13f460 2024-03-19T01:23:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22d2ffd640 1 -- 172.21.15.7:0/2803058608 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22d80b4420 con 0x7f22dc13f460 2024-03-19T01:23:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22dc1355d0 con 0x7f22dc13f460 2024-03-19T01:23:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22d2ffd640 1 -- 172.21.15.7:0/2803058608 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d80b4810 con 0x7f22dc13f460 2024-03-19T01:23:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22dc1429b0 con 0x7f22dc13f460 2024-03-19T01:23:47.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.507+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22dc075460 con 0x7f22dc13f460 2024-03-19T01:23:47.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.511+0000 7f22d2ffd640 1 -- 172.21.15.7:0/2803058608 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f22d80b49b0 con 0x7f22dc13f460 2024-03-19T01:23:47.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.511+0000 7f22d2ffd640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f22c0067660 0x7f22c0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.511+0000 7f22d2ffd640 1 -- 172.21.15.7:0/2803058608 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f22d811f4c0 con 0x7f22dc13f460 2024-03-19T01:23:47.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.511+0000 7f22e230e640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f22c0067660 0x7f22c0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.511+0000 7f22e230e640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f22c0067660 0x7f22c0069b20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f22dc142f60 tx=0x7f22cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.519+0000 7f22d2ffd640 1 -- 172.21.15.7:0/2803058608 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f22d8016110 con 0x7f22dc13f460 2024-03-19T01:23:47.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.703+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22dc074450 con 0x7f22dc13f460 2024-03-19T01:23:47.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22d2ffd640 1 -- 172.21.15.7:0/2803058608 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7f22d80f01d0 con 0x7f22dc13f460 2024-03-19T01:23:47.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:47.708 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f22c0067660 msgr2=0x7f22c0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f22c0067660 0x7f22c0069b20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f22dc142f60 tx=0x7f22cc008040 comp rx=0 tx=0).stop 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc13f460 msgr2=0x7f22dc1dde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc13f460 0x7f22dc1dde60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f22d801cd30 tx=0x7f22d8019230 comp rx=0 tx=0).stop 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 shutdown_connections 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f22c0067660 0x7f22c0069b20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22dc148ed0 0x7f22dc1de3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22dc13f460 0x7f22dc1dde60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 --2- 172.21.15.7:0/2803058608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22dc06fe80 0x7f22dc1446d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 >> 172.21.15.7:0/2803058608 conn(0x7f22dc0b5c20 msgr2=0x7f22dc1459f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 shutdown_connections 2024-03-19T01:23:47.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.707+0000 7f22e4572640 1 -- 172.21.15.7:0/2803058608 wait complete. 2024-03-19T01:23:47.727 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:47.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:47.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.859+0000 7fbd14361640 1 Processor -- start 2024-03-19T01:23:47.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.859+0000 7fbd14361640 1 -- start start 2024-03-19T01:23:47.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd14361640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c06b520 0x7fbd0c0af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd14361640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fbd0c0b4ce0 0x7fbd0c14b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd14361640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd0c0b6280 con 0x7fbd0c0af800 2024-03-19T01:23:47.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd14361640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fbd0c06b900 con 0x7fbd0c0b4ce0 2024-03-19T01:23:47.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd14361640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbd0c0b54e0 con 0x7fbd0c06b520 2024-03-19T01:23:47.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd118fc640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbd0c0af800 0x7fbd0c14ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35056/0 (socket says 172.21.15.7:35056) 2024-03-19T01:23:47.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd118fc640 1 -- 172.21.15.7:0/3758791929 learned_addr learned my addr 172.21.15.7:0/3758791929 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:47.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd120fd640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c06b520 0x7fbd0c0af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd128fe640 1 --2- 172.21.15.7:0/3758791929 >> v2:172.21.15.167:3300/0 conn(0x7fbd0c0b4ce0 0x7fbd0c14b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1405938349 0 0) 0x7fbcfc001120 con 0x7fbd0c0af800 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcf8003530 con 0x7fbd0c0af800 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4112984945 0 0) 0x7fbcf8003530 con 0x7fbd0c0af800 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c06b520 msgr2=0x7fbd0c0af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c06b520 0x7fbd0c0af2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 >> v2:172.21.15.167:3300/0 conn(0x7fbd0c0b4ce0 msgr2=0x7fbd0c14b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 --2- 172.21.15.7:0/3758791929 >> v2:172.21.15.167:3300/0 conn(0x7fbd0c0b4ce0 0x7fbd0c14b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0c0b5200 con 0x7fbd0c0af800 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fbcfc001850 con 0x7fbd0c0af800 2024-03-19T01:23:47.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 >> v1:172.21.15.7:6789/0 conn(0x7fbd0c0af800 legacy=0x7fbd0c14ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf80039c0 0x7fbcf8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcf8004810 0x7fbcf8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c06b520 0x7fbcf80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c06b900 con 0x7fbcf8004810 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c0b54e0 con 0x7fbcf80039c0 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf800ba50 con 0x7fbd0c06b520 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd118fc640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcf8004810 0x7fbcf8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd120fd640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf80039c0 0x7fbcf8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd128fe640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c06b520 0x7fbcf80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd118fc640 1 -- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c06b520 msgr2=0x7fbcf80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd118fc640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c06b520 0x7fbcf80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd118fc640 1 -- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf80039c0 msgr2=0x7fbcf8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:47.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd118fc640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf80039c0 0x7fbcf8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd118fc640 1 -- 172.21.15.7:0/3758791929 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf8003530 con 0x7fbcf8004810 2024-03-19T01:23:47.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.863+0000 7fbd128fe640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c06b520 0x7fbcf80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcf8004810 0x7fbcf8004c90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00c4c0 tx=0x7fbcfc01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4451937afe0ddf3b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00c0e0 con 0x7fbcf8004810 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcfc0a9020 con 0x7fbcf8004810 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd110fb640 1 -- 172.21.15.7:0/3758791929 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc005b20 con 0x7fbcf8004810 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- 172.21.15.7:0/3758791929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcf8004810 msgr2=0x7fbcf8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 --2- 172.21.15.7:0/3758791929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcf8004810 0x7fbcf8004c90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00c4c0 tx=0x7fbcfc01cca0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd128fe640 1 -- 172.21.15.7:0/3758791929 reap_dead start 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- 172.21.15.7:0/3758791929 shutdown_connections 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- 172.21.15.7:0/3758791929 >> 172.21.15.7:0/3758791929 conn(0x7fbd0c06e1b0 msgr2=0x7fbd0c06e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- 172.21.15.7:0/3758791929 shutdown_connections 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- 172.21.15.7:0/3758791929 wait complete. 2024-03-19T01:23:47.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 Processor -- start 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- start start 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c1417d0 0x7fbd0c1e0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c141bb0 0x7fbd0c1e0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd0c1e34c0 0x7fbd0c1e3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c14b900 con 0x7fbd0c141bb0 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c14b780 con 0x7fbd0c1e34c0 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbd0c1528f0 con 0x7fbd0c1417d0 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c141bb0 0x7fbd0c1e0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c141bb0 0x7fbd0c1e0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42890/0 (socket says 172.21.15.7:42890) 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 -- 172.21.15.7:0/4057497622 learned_addr learned my addr 172.21.15.7:0/4057497622 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd128fe640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd0c1e34c0 0x7fbd0c1e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd120fd640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c1417d0 0x7fbd0c1e0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 -- 172.21.15.7:0/4057497622 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c1417d0 msgr2=0x7fbd0c1e0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c1417d0 0x7fbd0c1e0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 -- 172.21.15.7:0/4057497622 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd0c1e34c0 msgr2=0x7fbd0c1e3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd0c1e34c0 0x7fbd0c1e3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 -- 172.21.15.7:0/4057497622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0c14faa0 con 0x7fbd0c141bb0 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd118fc640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c141bb0 0x7fbd0c1e0870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0ab040 tx=0x7fbcf8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd02ffd640 1 -- 172.21.15.7:0/4057497622 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc0048b0 con 0x7fbd0c141bb0 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd02ffd640 1 -- 172.21.15.7:0/4057497622 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcfc004a50 con 0x7fbd0c141bb0 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0c0b4140 con 0x7fbd0c141bb0 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd02ffd640 1 -- 172.21.15.7:0/4057497622 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc004e40 con 0x7fbd0c141bb0 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.867+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd0c141050 con 0x7fbd0c141bb0 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.871+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd0c075570 con 0x7fbd0c141bb0 2024-03-19T01:23:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.871+0000 7fbd02ffd640 1 -- 172.21.15.7:0/4057497622 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbcfc0bd060 con 0x7fbd0c141bb0 2024-03-19T01:23:47.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.871+0000 7fbd02ffd640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbcf0067690 0x7fbcf0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:47.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.871+0000 7fbd02ffd640 1 -- 172.21.15.7:0/4057497622 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbcfc11e350 con 0x7fbd0c141bb0 2024-03-19T01:23:47.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.871+0000 7fbd120fd640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbcf0067690 0x7fbcf0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:47.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.871+0000 7fbd120fd640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbcf0067690 0x7fbcf0069b50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbd040099e0 tx=0x7fbd04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:47.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:47.875+0000 7fbd02ffd640 1 -- 172.21.15.7:0/4057497622 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbd0c0b6280 con 0x7fbd0c141bb0 2024-03-19T01:23:48.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.115+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbd0c0710a0 con 0x7fbd0c141bb0 2024-03-19T01:23:48.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.115+0000 7fbd02ffd640 1 -- 172.21.15.7:0/4057497622 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fbcfc0ef060 con 0x7fbd0c141bb0 2024-03-19T01:23:48.118 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbcf0067690 msgr2=0x7fbcf0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbcf0067690 0x7fbcf0069b50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbd040099e0 tx=0x7fbd04008040 comp rx=0 tx=0).stop 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c141bb0 msgr2=0x7fbd0c1e0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c141bb0 0x7fbd0c1e0870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0ab040 tx=0x7fbcf8005840 comp rx=0 tx=0).stop 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 shutdown_connections 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fbcf0067690 0x7fbcf0069b50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd0c1e34c0 0x7fbd0c1e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c141bb0 0x7fbd0c1e0870 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 --2- 172.21.15.7:0/4057497622 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd0c1417d0 0x7fbd0c1e0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 >> 172.21.15.7:0/4057497622 conn(0x7fbd0c06e1b0 msgr2=0x7fbd0c06e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 shutdown_connections 2024-03-19T01:23:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.119+0000 7fbd14361640 1 -- 172.21.15.7:0/4057497622 wait complete. 2024-03-19T01:23:48.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f0208cb6640 1 Processor -- start 2024-03-19T01:23:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f0208cb6640 1 -- start start 2024-03-19T01:23:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f0208cb6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02040b59a0 0x7f02040ae910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f0208cb6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02040add50 0x7f02040b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f0208cb6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02040b5d80 con 0x7f02040af5d0 2024-03-19T01:23:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f0208cb6640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f020406cad0 con 0x7f02040b59a0 2024-03-19T01:23:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f0208cb6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f02040ae3f0 con 0x7f02040add50 2024-03-19T01:23:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f020259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02040b59a0 0x7f02040ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f020259c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02040b59a0 0x7f02040ae910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42896/0 (socket says 172.21.15.7:42896) 2024-03-19T01:23:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f020259c640 1 -- 172.21.15.7:0/2503485252 learned_addr learned my addr 172.21.15.7:0/2503485252 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f0201d9b640 1 --2- 172.21.15.7:0/2503485252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02040add50 0x7f02040b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f020259c640 1 -- 172.21.15.7:0/2503485252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02040add50 msgr2=0x7f02040b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f020259c640 1 --2- 172.21.15.7:0/2503485252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02040add50 0x7f02040b1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f020259c640 1 -- 172.21.15.7:0/2503485252 >> v1:172.21.15.167:6789/0 conn(0x7f02040af5d0 legacy=0x7f02040b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.255+0000 7f020259c640 1 -- 172.21.15.7:0/2503485252 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02040b4eb0 con 0x7f02040b59a0 2024-03-19T01:23:48.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f020259c640 1 --2- 172.21.15.7:0/2503485252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040b59a0 0x7f02040ae910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009870 tx=0x7f01f401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900a866885d4b300 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:48.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f020159a640 1 -- 172.21.15.7:0/2503485252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f40b0070 con 0x7f02040b59a0 2024-03-19T01:23:48.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f020159a640 1 -- 172.21.15.7:0/2503485252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01f40ab600 con 0x7f02040b59a0 2024-03-19T01:23:48.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f020159a640 1 -- 172.21.15.7:0/2503485252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f40ab9d0 con 0x7f02040b59a0 2024-03-19T01:23:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 -- 172.21.15.7:0/2503485252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040b59a0 msgr2=0x7f02040ae910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 --2- 172.21.15.7:0/2503485252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040b59a0 0x7f02040ae910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009870 tx=0x7f01f401c490 comp rx=0 tx=0).stop 2024-03-19T01:23:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 -- 172.21.15.7:0/2503485252 shutdown_connections 2024-03-19T01:23:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 --2- 172.21.15.7:0/2503485252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02040add50 0x7f02040b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 --2- 172.21.15.7:0/2503485252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040b59a0 0x7f02040ae910 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 -- 172.21.15.7:0/2503485252 >> 172.21.15.7:0/2503485252 conn(0x7f020406bfb0 msgr2=0x7f020406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 -- 172.21.15.7:0/2503485252 shutdown_connections 2024-03-19T01:23:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 -- 172.21.15.7:0/2503485252 wait complete. 2024-03-19T01:23:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 Processor -- start 2024-03-19T01:23:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.259+0000 7f0208cb6640 1 -- start start 2024-03-19T01:23:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0208cb6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040af5d0 0x7f0204147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0208cb6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02040b59a0 0x7f0204147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0208cb6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02041e0350 0x7f02041520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0208cb6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f02041551e0 con 0x7f02040af5d0 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0208cb6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0204155060 con 0x7f02041e0350 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0208cb6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0204155360 con 0x7f02040b59a0 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f020259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040af5d0 0x7f0204147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f020259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040af5d0 0x7f0204147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42908/0 (socket says 172.21.15.7:42908) 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f020259c640 1 -- 172.21.15.7:0/3732745891 learned_addr learned my addr 172.21.15.7:0/3732745891 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f020259c640 1 -- 172.21.15.7:0/3732745891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02040b59a0 msgr2=0x7f0204147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f020259c640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02040b59a0 0x7f0204147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f020259c640 1 -- 172.21.15.7:0/3732745891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02041e0350 msgr2=0x7f02041520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f020259c640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02041e0350 0x7f02041520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f020259c640 1 -- 172.21.15.7:0/3732745891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020414c3b0 con 0x7f02040af5d0 2024-03-19T01:23:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f020259c640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040af5d0 0x7f0204147180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01f40ae040 tx=0x7f01f401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f01f2ffd640 1 -- 172.21.15.7:0/3732745891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f40ba020 con 0x7f02040af5d0 2024-03-19T01:23:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f01f2ffd640 1 -- 172.21.15.7:0/3732745891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01f40b6620 con 0x7f02040af5d0 2024-03-19T01:23:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020414c690 con 0x7f02040af5d0 2024-03-19T01:23:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f01f2ffd640 1 -- 172.21.15.7:0/3732745891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f40b0040 con 0x7f02040af5d0 2024-03-19T01:23:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0204146980 con 0x7f02040af5d0 2024-03-19T01:23:48.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0204075570 con 0x7f02040af5d0 2024-03-19T01:23:48.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f01f2ffd640 1 -- 172.21.15.7:0/3732745891 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f01f40bd020 con 0x7f02040af5d0 2024-03-19T01:23:48.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f01f2ffd640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01e4067660 0x7f01e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0201d9b640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01e4067660 0x7f01e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f01f2ffd640 1 -- 172.21.15.7:0/3732745891 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f01f411fce0 con 0x7f02040af5d0 2024-03-19T01:23:48.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.263+0000 7f0201d9b640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01e4067660 0x7f01e4069b20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f01f8006fd0 tx=0x7f01f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.267+0000 7f01f2ffd640 1 -- 172.21.15.7:0/3732745891 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f01f40c4030 con 0x7f02040af5d0 2024-03-19T01:23:48.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.463+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f020406f530 con 0x7f02040af5d0 2024-03-19T01:23:48.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.463+0000 7f01f2ffd640 1 -- 172.21.15.7:0/3732745891 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f01f4016110 con 0x7f02040af5d0 2024-03-19T01:23:48.464 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.463+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01e4067660 msgr2=0x7f01e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.463+0000 7f0208cb6640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01e4067660 0x7f01e4069b20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f01f8006fd0 tx=0x7f01f8008040 comp rx=0 tx=0).stop 2024-03-19T01:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040af5d0 msgr2=0x7f0204147180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040af5d0 0x7f0204147180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01f40ae040 tx=0x7f01f401acd0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 shutdown_connections 2024-03-19T01:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01e4067660 0x7f01e4069b20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02040b59a0 0x7f0204147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02041e0350 0x7f02041520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 --2- 172.21.15.7:0/3732745891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02040af5d0 0x7f0204147180 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 >> 172.21.15.7:0/3732745891 conn(0x7f020406bfb0 msgr2=0x7f0204067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:48.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 shutdown_connections 2024-03-19T01:23:48.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.467+0000 7f0208cb6640 1 -- 172.21.15.7:0/3732745891 wait complete. 2024-03-19T01:23:48.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.611+0000 7f0cd0aca640 1 Processor -- start 2024-03-19T01:23:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cd0aca640 1 -- start start 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cd0aca640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc0b56e0 0x7f0ccc0b4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cd0aca640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ccc0b2c20 con 0x7f0ccc0b48c0 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cd0aca640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ccc0b5d00 con 0x7f0ccc0b77f0 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cd0aca640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0ccc0b5fe0 con 0x7f0ccc0b56e0 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0ccad9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0ccc0b48c0 0x7f0ccc14fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35072/0 (socket says 172.21.15.7:35072) 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0ccad9d640 1 -- 172.21.15.7:0/3854885690 learned_addr learned my addr 172.21.15.7:0/3854885690 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc3fff640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc0b56e0 0x7f0ccc0b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151751031 0 0) 0x7f0ccc0b2c20 con 0x7f0ccc0b48c0 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cb8003530 con 0x7f0ccc0b48c0 2024-03-19T01:23:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2787160763 0 0) 0x7f0cb8003530 con 0x7f0ccc0b48c0 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc0b56e0 msgr2=0x7f0ccc0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc0b56e0 0x7f0ccc0b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 >> v1:172.21.15.167:6789/0 conn(0x7f0ccc0b77f0 legacy=0x7f0ccc0b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ccc0b64d0 con 0x7f0ccc0b48c0 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f0cc4001a00 con 0x7f0ccc0b48c0 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 >> v1:172.21.15.7:6789/0 conn(0x7f0ccc0b48c0 legacy=0x7f0ccc14fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cb80039f0 0x7f0cb8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc0b56e0 0x7f0cb8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cb8004d50 0x7f0cb80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc0b5fe0 con 0x7f0cb8004d50 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb800bb30 con 0x7f0cb80039f0 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0cb800bd30 con 0x7f0ccc0b56e0 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0ccad9d640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cb8004d50 0x7f0cb80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc3fff640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc0b56e0 0x7f0cb8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cca59c640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cb80039f0 0x7f0cb8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0ccad9d640 1 -- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc0b56e0 msgr2=0x7f0cb8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0ccad9d640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc0b56e0 0x7f0cb8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0ccad9d640 1 -- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cb80039f0 msgr2=0x7f0cb8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0ccad9d640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cb80039f0 0x7f0cb8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0ccad9d640 1 -- 172.21.15.7:0/3854885690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb8003530 con 0x7f0cb8004d50 2024-03-19T01:23:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cc3fff640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc0b56e0 0x7f0cb8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.615+0000 7f0cca59c640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cb80039f0 0x7f0cb8003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0ccad9d640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cb8004d50 0x7f0cb80051d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400a860 tx=0x7f0cc401fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949b9966147bed76 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc400a480 con 0x7f0cb8004d50 2024-03-19T01:23:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc40aa710 con 0x7f0cb8004d50 2024-03-19T01:23:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cc9d9b640 1 -- 172.21.15.7:0/3854885690 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc40aab20 con 0x7f0cb8004d50 2024-03-19T01:23:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cd0aca640 1 -- 172.21.15.7:0/3854885690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cb8004d50 msgr2=0x7f0cb80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cd0aca640 1 --2- 172.21.15.7:0/3854885690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cb8004d50 0x7f0cb80051d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400a860 tx=0x7f0cc401fae0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0ccad9d640 1 -- 172.21.15.7:0/3854885690 reap_dead start 2024-03-19T01:23:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cd0aca640 1 -- 172.21.15.7:0/3854885690 shutdown_connections 2024-03-19T01:23:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cd0aca640 1 -- 172.21.15.7:0/3854885690 >> 172.21.15.7:0/3854885690 conn(0x7f0ccc06b2c0 msgr2=0x7f0ccc06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cd0aca640 1 -- 172.21.15.7:0/3854885690 shutdown_connections 2024-03-19T01:23:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cd0aca640 1 -- 172.21.15.7:0/3854885690 wait complete. 2024-03-19T01:23:48.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cd0aca640 1 Processor -- start 2024-03-19T01:23:48.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.619+0000 7f0cd0aca640 1 -- start start 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cd0aca640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc1419f0 0x7f0ccc142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cd0aca640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ccc1426a0 0x7f0ccc1dae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cd0aca640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ccc1db3b0 0x7f0ccc1ddee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cd0aca640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc072bb0 con 0x7f0ccc1426a0 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cd0aca640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc0b7bd0 con 0x7f0ccc1db3b0 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cd0aca640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0ccc072d30 con 0x7f0ccc1419f0 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ccc1426a0 0x7f0ccc1dae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ccc1426a0 0x7f0ccc1dae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42924/0 (socket says 172.21.15.7:42924) 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc3fff640 1 -- 172.21.15.7:0/561394654 learned_addr learned my addr 172.21.15.7:0/561394654 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cca59c640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc1419f0 0x7f0ccc142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0ccad9d640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ccc1db3b0 0x7f0ccc1ddee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc3fff640 1 -- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc1419f0 msgr2=0x7f0ccc142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc3fff640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc1419f0 0x7f0ccc142160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc3fff640 1 -- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ccc1db3b0 msgr2=0x7f0ccc1ddee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc3fff640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ccc1db3b0 0x7f0ccc1ddee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc3fff640 1 -- 172.21.15.7:0/561394654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ccc1da9b0 con 0x7f0ccc1426a0 2024-03-19T01:23:48.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cca59c640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc1419f0 0x7f0ccc142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:48.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0ccad9d640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ccc1db3b0 0x7f0ccc1ddee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:48.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc3fff640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ccc1426a0 0x7f0ccc1dae70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40116a0 tx=0x7f0cb4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:48.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc2ffd640 1 -- 172.21.15.7:0/561394654 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4004220 con 0x7f0ccc1426a0 2024-03-19T01:23:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc2ffd640 1 -- 172.21.15.7:0/561394654 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb40043c0 con 0x7f0ccc1426a0 2024-03-19T01:23:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ccc142b00 con 0x7f0ccc1426a0 2024-03-19T01:23:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc2ffd640 1 -- 172.21.15.7:0/561394654 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb400f980 con 0x7f0ccc1426a0 2024-03-19T01:23:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ccc140700 con 0x7f0ccc1426a0 2024-03-19T01:23:48.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ccc075570 con 0x7f0ccc1426a0 2024-03-19T01:23:48.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc2ffd640 1 -- 172.21.15.7:0/561394654 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0cb400fb20 con 0x7f0ccc1426a0 2024-03-19T01:23:48.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc2ffd640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0cac067660 0x7f0cac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:48.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.623+0000 7f0cc2ffd640 1 -- 172.21.15.7:0/561394654 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0cb4086270 con 0x7f0ccc1426a0 2024-03-19T01:23:48.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.627+0000 7f0cca59c640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0cac067660 0x7f0cac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:48.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.627+0000 7f0cca59c640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0cac067660 0x7f0cac069b20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0016e0 tx=0x7f0cbc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:48.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.627+0000 7f0cc2ffd640 1 -- 172.21.15.7:0/561394654 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0ccc142b00 con 0x7f0ccc1426a0 2024-03-19T01:23:48.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0ccc13e960 con 0x7f0ccc1426a0 2024-03-19T01:23:48.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cc2ffd640 1 -- 172.21.15.7:0/561394654 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f0ccc1da9b0 con 0x7f0ccc1426a0 2024-03-19T01:23:48.856 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0cac067660 msgr2=0x7f0cac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0cac067660 0x7f0cac069b20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0016e0 tx=0x7f0cbc009040 comp rx=0 tx=0).stop 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ccc1426a0 msgr2=0x7f0ccc1dae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ccc1426a0 0x7f0ccc1dae70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40116a0 tx=0x7f0cb4011b60 comp rx=0 tx=0).stop 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 shutdown_connections 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0cac067660 0x7f0cac069b20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ccc1db3b0 0x7f0ccc1ddee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ccc1426a0 0x7f0ccc1dae70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 --2- 172.21.15.7:0/561394654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ccc1419f0 0x7f0ccc142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:48.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 >> 172.21.15.7:0/561394654 conn(0x7f0ccc06b2c0 msgr2=0x7f0ccc0714a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:48.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 shutdown_connections 2024-03-19T01:23:48.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:48.855+0000 7f0cd0aca640 1 -- 172.21.15.7:0/561394654 wait complete. 2024-03-19T01:23:48.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29c75d640 1 Processor -- start 2024-03-19T01:23:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29c75d640 1 -- start start 2024-03-19T01:23:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29c75d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fc2940726b0 0x7fc294148940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29c75d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc294072a90 con 0x7fc2940b3a20 2024-03-19T01:23:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29c75d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2940b4a30 con 0x7fc2940ba710 2024-03-19T01:23:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29c75d640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fc294148e80 con 0x7fc2940726b0 2024-03-19T01:23:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29acfa640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc2940b3a20 0x7fc29414fc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35074/0 (socket says 172.21.15.7:35074) 2024-03-19T01:23:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29acfa640 1 -- 172.21.15.7:0/889718415 learned_addr learned my addr 172.21.15.7:0/889718415 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 567664686 0 0) 0x7fc294072a90 con 0x7fc2940b3a20 2024-03-19T01:23:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29a4f9640 1 --2- 172.21.15.7:0/889718415 >> v2:172.21.15.167:3300/0 conn(0x7fc2940726b0 0x7fc294148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc280003530 con 0x7fc2940b3a20 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2378958410 0 0) 0x7fc280003530 con 0x7fc2940b3a20 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 >> v2:172.21.15.167:3300/0 conn(0x7fc2940726b0 msgr2=0x7fc294148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 --2- 172.21.15.7:0/889718415 >> v2:172.21.15.167:3300/0 conn(0x7fc2940726b0 0x7fc294148940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 >> v1:172.21.15.167:6789/0 conn(0x7fc2940ba710 legacy=0x7fc2940b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2940b4750 con 0x7fc2940b3a20 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fc28c001a00 con 0x7fc2940b3a20 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 >> v1:172.21.15.7:6789/0 conn(0x7fc2940b3a20 legacy=0x7fc29414fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2800039c0 0x7fc280003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc280004810 0x7fc280004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2800051d0 0x7fc28000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc294148e80 con 0x7fc2800051d0 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc28000c100 con 0x7fc2800039c0 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc28000c300 con 0x7fc280004810 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29acfa640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2800051d0 0x7fc28000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29a4f9640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2800039c0 0x7fc280003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc299cf8640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc280004810 0x7fc280004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29acfa640 1 -- 172.21.15.7:0/889718415 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc280004810 msgr2=0x7fc280004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29acfa640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc280004810 0x7fc280004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29acfa640 1 -- 172.21.15.7:0/889718415 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2800039c0 msgr2=0x7fc280003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29acfa640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2800039c0 0x7fc280003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.003+0000 7fc29acfa640 1 -- 172.21.15.7:0/889718415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc280003530 con 0x7fc2800051d0 2024-03-19T01:23:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 -- 172.21.15.7:0/889718415 reap_dead start 2024-03-19T01:23:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2800051d0 0x7fc28000b9d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00a880 tx=0x7fc28c01fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793392e86d19c5d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c00a4a0 con 0x7fc2800051d0 2024-03-19T01:23:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc28c0aa510 con 0x7fc2800051d0 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc2994f7640 1 -- 172.21.15.7:0/889718415 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0aa8e0 con 0x7fc2800051d0 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 -- 172.21.15.7:0/889718415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2800051d0 msgr2=0x7fc28000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2800051d0 0x7fc28000b9d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00a880 tx=0x7fc28c01fae0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 -- 172.21.15.7:0/889718415 shutdown_connections 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 --2- 172.21.15.7:0/889718415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2800051d0 0x7fc28000b9d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 -- 172.21.15.7:0/889718415 >> 172.21.15.7:0/889718415 conn(0x7fc294074130 msgr2=0x7fc294074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 -- 172.21.15.7:0/889718415 shutdown_connections 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 -- 172.21.15.7:0/889718415 wait complete. 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 Processor -- start 2024-03-19T01:23:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 -- start start 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2800051d0 0x7fc2941480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2941dfa30 0x7fc294070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc294071bc0 0x7fc2941e0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc2940b44c0 con 0x7fc294071bc0 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc2940b5750 con 0x7fc2941dfa30 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29c75d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc2940b0bb0 con 0x7fc2800051d0 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc294071bc0 0x7fc2941e0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc294071bc0 0x7fc2941e0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42948/0 (socket says 172.21.15.7:42948) 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 -- 172.21.15.7:0/3974783234 learned_addr learned my addr 172.21.15.7:0/3974783234 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29a4f9640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2800051d0 0x7fc2941480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc299cf8640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2941dfa30 0x7fc294070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 -- 172.21.15.7:0/3974783234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2800051d0 msgr2=0x7fc2941480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2800051d0 0x7fc2941480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 -- 172.21.15.7:0/3974783234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2941dfa30 msgr2=0x7fc294070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2941dfa30 0x7fc294070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 -- 172.21.15.7:0/3974783234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc294070400 con 0x7fc294071bc0 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc299cf8640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2941dfa30 0x7fc294070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.007+0000 7fc29acfa640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc294071bc0 0x7fc2941e0040 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc28c020380 tx=0x7fc280005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc28affd640 1 -- 172.21.15.7:0/3974783234 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c00a4a0 con 0x7fc294071bc0 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc28affd640 1 -- 172.21.15.7:0/3974783234 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc28c0bb070 con 0x7fc294071bc0 2024-03-19T01:23:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc294145960 con 0x7fc294071bc0 2024-03-19T01:23:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc28affd640 1 -- 172.21.15.7:0/3974783234 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0aa900 con 0x7fc294071bc0 2024-03-19T01:23:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2940b0830 con 0x7fc294071bc0 2024-03-19T01:23:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc294075460 con 0x7fc294071bc0 2024-03-19T01:23:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc28affd640 1 -- 172.21.15.7:0/3974783234 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc28c0b3040 con 0x7fc294071bc0 2024-03-19T01:23:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc28affd640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc278067660 0x7fc278069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc28affd640 1 -- 172.21.15.7:0/3974783234 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc28c122b50 con 0x7fc294071bc0 2024-03-19T01:23:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc29a4f9640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc278067660 0x7fc278069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.011+0000 7fc29a4f9640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc278067660 0x7fc278069b20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc290009910 tx=0x7fc290008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.015+0000 7fc28affd640 1 -- 172.21.15.7:0/3974783234 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc28c0197f0 con 0x7fc294071bc0 2024-03-19T01:23:49.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.175+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2940b4a30 con 0x7fc294071bc0 2024-03-19T01:23:49.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc28affd640 1 -- 172.21.15.7:0/3974783234 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7fc28c0f3860 con 0x7fc294071bc0 2024-03-19T01:23:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:49.179 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc278067660 msgr2=0x7fc278069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc278067660 0x7fc278069b20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc290009910 tx=0x7fc290008040 comp rx=0 tx=0).stop 2024-03-19T01:23:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc294071bc0 msgr2=0x7fc2941e0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc294071bc0 0x7fc2941e0040 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc28c020380 tx=0x7fc280005840 comp rx=0 tx=0).stop 2024-03-19T01:23:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 shutdown_connections 2024-03-19T01:23:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc278067660 0x7fc278069b20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc294071bc0 0x7fc2941e0040 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2941dfa30 0x7fc294070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 --2- 172.21.15.7:0/3974783234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2800051d0 0x7fc2941480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 >> 172.21.15.7:0/3974783234 conn(0x7fc294074130 msgr2=0x7fc2940b8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:49.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 shutdown_connections 2024-03-19T01:23:49.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.179+0000 7fc29c75d640 1 -- 172.21.15.7:0/3974783234 wait complete. 2024-03-19T01:23:49.196 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:49.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:49.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.307+0000 7fae68a51640 1 Processor -- start 2024-03-19T01:23:49.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.307+0000 7fae68a51640 1 -- start start 2024-03-19T01:23:49.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.307+0000 7fae68a51640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fae6406b520 0x7fae640b5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.307+0000 7fae68a51640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae640aeee0 con 0x7fae640725d0 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.307+0000 7fae68a51640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae6406b900 con 0x7fae640ba080 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.307+0000 7fae68a51640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fae6409f260 con 0x7fae6406b520 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae61d9b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fae640725d0 0x7fae640b9970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35082/0 (socket says 172.21.15.7:35082) 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae61d9b640 1 -- 172.21.15.7:0/2789825057 learned_addr learned my addr 172.21.15.7:0/2789825057 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6259c640 1 --2- 172.21.15.7:0/2789825057 >> v2:172.21.15.167:3300/0 conn(0x7fae6406b520 0x7fae640b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240861550 0 0) 0x7fae54001120 con 0x7fae640725d0 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae4c003530 con 0x7fae640725d0 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4047316426 0 0) 0x7fae4c003530 con 0x7fae640725d0 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 >> v2:172.21.15.167:3300/0 conn(0x7fae6406b520 msgr2=0x7fae640b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 --2- 172.21.15.7:0/2789825057 >> v2:172.21.15.167:3300/0 conn(0x7fae6406b520 0x7fae640b5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 >> v1:172.21.15.167:6789/0 conn(0x7fae640ba080 legacy=0x7fae64151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae640b5570 con 0x7fae640725d0 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fae54001850 con 0x7fae640725d0 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 >> v1:172.21.15.7:6789/0 conn(0x7fae640725d0 legacy=0x7fae640b9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae4c0039c0 0x7fae4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c004810 0x7fae4c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae4c0052f0 0x7fae4c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fae6409f260 con 0x7fae4c0039c0 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c00c1c0 con 0x7fae4c0052f0 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fae4c00c3c0 con 0x7fae4c004810 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6259c640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae4c0039c0 0x7fae4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae61d9b640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c004810 0x7fae4c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae62d9d640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae4c0052f0 0x7fae4c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6259c640 1 -- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c004810 msgr2=0x7fae4c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6259c640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c004810 0x7fae4c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6259c640 1 -- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae4c0052f0 msgr2=0x7fae4c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6259c640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae4c0052f0 0x7fae4c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6259c640 1 -- 172.21.15.7:0/2789825057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4c003530 con 0x7fae4c0039c0 2024-03-19T01:23:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae62d9d640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae4c0052f0 0x7fae4c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae61d9b640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c004810 0x7fae4c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae62d9d640 1 -- 172.21.15.7:0/2789825057 reap_dead start 2024-03-19T01:23:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6259c640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae4c0039c0 0x7fae4c003de0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fae58009ab0 tx=0x7fae5801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62be9c2f950721c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae580b2070 con 0x7fae4c0039c0 2024-03-19T01:23:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae580a5640 con 0x7fae4c0039c0 2024-03-19T01:23:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.311+0000 7fae6159a640 1 -- 172.21.15.7:0/2789825057 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae580096b0 con 0x7fae4c0039c0 2024-03-19T01:23:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- 172.21.15.7:0/2789825057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae4c0039c0 msgr2=0x7fae4c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae4c0039c0 0x7fae4c003de0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fae58009ab0 tx=0x7fae5801ca50 comp rx=0 tx=0).stop 2024-03-19T01:23:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- 172.21.15.7:0/2789825057 shutdown_connections 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 --2- 172.21.15.7:0/2789825057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae4c0039c0 0x7fae4c003de0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- 172.21.15.7:0/2789825057 >> 172.21.15.7:0/2789825057 conn(0x7fae6406e1b0 msgr2=0x7fae6406e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- 172.21.15.7:0/2789825057 shutdown_connections 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- 172.21.15.7:0/2789825057 wait complete. 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 Processor -- start 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- start start 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c0039c0 0x7fae641d62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae6414a480 0x7fae6414a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae641e36b0 0x7fae641e3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fae6409fc90 con 0x7fae641e36b0 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fae640b5f90 con 0x7fae6414a480 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fae6409fe10 con 0x7fae4c0039c0 2024-03-19T01:23:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae62d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae641e36b0 0x7fae641e3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae62d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae641e36b0 0x7fae641e3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42960/0 (socket says 172.21.15.7:42960) 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae62d9d640 1 -- 172.21.15.7:0/3234558590 learned_addr learned my addr 172.21.15.7:0/3234558590 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae6259c640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c0039c0 0x7fae641d62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae62d9d640 1 -- 172.21.15.7:0/3234558590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c0039c0 msgr2=0x7fae641d62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae62d9d640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c0039c0 0x7fae641d62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae62d9d640 1 -- 172.21.15.7:0/3234558590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae6414a480 msgr2=0x7fae6414a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae61d9b640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae6414a480 0x7fae6414a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae62d9d640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae6414a480 0x7fae6414a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae62d9d640 1 -- 172.21.15.7:0/3234558590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae64148090 con 0x7fae641e36b0 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae62d9d640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae641e36b0 0x7fae641e3b30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fae6406f200 tx=0x7fae5c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae52ffd640 1 -- 172.21.15.7:0/3234558590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c018070 con 0x7fae641e36b0 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae52ffd640 1 -- 172.21.15.7:0/3234558590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5c003ba0 con 0x7fae641e36b0 2024-03-19T01:23:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae641485e0 con 0x7fae641e36b0 2024-03-19T01:23:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae52ffd640 1 -- 172.21.15.7:0/3234558590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c005070 con 0x7fae641e36b0 2024-03-19T01:23:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae64149d10 con 0x7fae641e36b0 2024-03-19T01:23:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.315+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae64075570 con 0x7fae641e36b0 2024-03-19T01:23:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.319+0000 7fae52ffd640 1 -- 172.21.15.7:0/3234558590 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fae5c01b070 con 0x7fae641e36b0 2024-03-19T01:23:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.319+0000 7fae52ffd640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fae44067660 0x7fae44069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.319+0000 7fae52ffd640 1 -- 172.21.15.7:0/3234558590 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fae5c084fe0 con 0x7fae641e36b0 2024-03-19T01:23:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.319+0000 7fae6259c640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fae44067660 0x7fae44069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.319+0000 7fae6259c640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fae44067660 0x7fae44069b20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fae580af000 tx=0x7fae58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.323+0000 7fae52ffd640 1 -- 172.21.15.7:0/3234558590 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fae5c055c70 con 0x7fae641e36b0 2024-03-19T01:23:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fae6406ca90 con 0x7fae641e36b0 2024-03-19T01:23:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae52ffd640 1 -- 172.21.15.7:0/3234558590 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fae5c055570 con 0x7fae641e36b0 2024-03-19T01:23:49.572 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fae44067660 msgr2=0x7fae44069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fae44067660 0x7fae44069b20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fae580af000 tx=0x7fae58008040 comp rx=0 tx=0).stop 2024-03-19T01:23:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae641e36b0 msgr2=0x7fae641e3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae641e36b0 0x7fae641e3b30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fae6406f200 tx=0x7fae5c00dc00 comp rx=0 tx=0).stop 2024-03-19T01:23:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 shutdown_connections 2024-03-19T01:23:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fae44067660 0x7fae44069b20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae641e36b0 0x7fae641e3b30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fae6414a480 0x7fae6414a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 --2- 172.21.15.7:0/3234558590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae4c0039c0 0x7fae641d62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 >> 172.21.15.7:0/3234558590 conn(0x7fae6406e1b0 msgr2=0x7fae640b6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 shutdown_connections 2024-03-19T01:23:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.571+0000 7fae68a51640 1 -- 172.21.15.7:0/3234558590 wait complete. 2024-03-19T01:23:49.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:49.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f276e16d640 1 Processor -- start 2024-03-19T01:23:49.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f276e16d640 1 -- start start 2024-03-19T01:23:49.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f276e16d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f27680b59a0 0x7f27681450d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f276e16d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f276806ff90 0x7f2768071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f276e16d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27680b5d80 con 0x7f2768072440 2024-03-19T01:23:49.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f276e16d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f2768145610 con 0x7f27680b59a0 2024-03-19T01:23:49.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f276e16d640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f27680704c0 con 0x7f276806ff90 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f27677fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f27680b59a0 0x7f27681450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f2767fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2768072440 0x7f27680b1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35086/0 (socket says 172.21.15.7:35086) 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f2767fff640 1 -- 172.21.15.7:0/768872588 learned_addr learned my addr 172.21.15.7:0/768872588 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f2766ffd640 1 --2- 172.21.15.7:0/768872588 >> v2:172.21.15.167:3300/0 conn(0x7f276806ff90 0x7f2768071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f27667fc640 1 -- 172.21.15.7:0/768872588 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2150437005 0 0) 0x7f27680b5d80 con 0x7f2768072440 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f27667fc640 1 -- 172.21.15.7:0/768872588 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2754003530 con 0x7f2768072440 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f27677fe640 1 -- 172.21.15.7:0/768872588 >> v2:172.21.15.167:3300/0 conn(0x7f276806ff90 msgr2=0x7f2768071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f27677fe640 1 --2- 172.21.15.7:0/768872588 >> v2:172.21.15.167:3300/0 conn(0x7f276806ff90 0x7f2768071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f27677fe640 1 -- 172.21.15.7:0/768872588 >> v1:172.21.15.7:6789/0 conn(0x7f2768072440 legacy=0x7f27680b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.723+0000 7f27677fe640 1 -- 172.21.15.7:0/768872588 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2768072a80 con 0x7f27680b59a0 2024-03-19T01:23:49.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f27677fe640 1 --2- 172.21.15.7:0/768872588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27680b59a0 0x7f27681450d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2768146260 tx=0x7f275801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750197b523193995 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:49.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f27667fc640 1 -- 172.21.15.7:0/768872588 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27580b0070 con 0x7f27680b59a0 2024-03-19T01:23:49.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f27667fc640 1 -- 172.21.15.7:0/768872588 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27580ab400 con 0x7f27680b59a0 2024-03-19T01:23:49.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f27667fc640 1 -- 172.21.15.7:0/768872588 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27580ab810 con 0x7f27680b59a0 2024-03-19T01:23:49.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- 172.21.15.7:0/768872588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27680b59a0 msgr2=0x7f27681450d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 --2- 172.21.15.7:0/768872588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27680b59a0 0x7f27681450d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2768146260 tx=0x7f275801c380 comp rx=0 tx=0).stop 2024-03-19T01:23:49.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- 172.21.15.7:0/768872588 shutdown_connections 2024-03-19T01:23:49.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 --2- 172.21.15.7:0/768872588 >> v2:172.21.15.167:3300/0 conn(0x7f276806ff90 0x7f2768071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 --2- 172.21.15.7:0/768872588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27680b59a0 0x7f27681450d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- 172.21.15.7:0/768872588 >> 172.21.15.7:0/768872588 conn(0x7f276806c3b0 msgr2=0x7f276806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- 172.21.15.7:0/768872588 shutdown_connections 2024-03-19T01:23:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- 172.21.15.7:0/768872588 wait complete. 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 Processor -- start 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- start start 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f276806ff90 0x7f27681e3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2768072440 0x7f27680addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27680b59a0 0x7f27680ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f27680b4f50 con 0x7f2768072440 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f27680b4dd0 con 0x7f27680b59a0 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2768152a70 con 0x7f276806ff90 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2766ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2768072440 0x7f27680addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2766ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2768072440 0x7f27680addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42982/0 (socket says 172.21.15.7:42982) 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2766ffd640 1 -- 172.21.15.7:0/2825772968 learned_addr learned my addr 172.21.15.7:0/2825772968 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f27677fe640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f276806ff90 0x7f27681e3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2767fff640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27680b59a0 0x7f27680ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2766ffd640 1 -- 172.21.15.7:0/2825772968 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f276806ff90 msgr2=0x7f27681e3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2766ffd640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f276806ff90 0x7f27681e3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2766ffd640 1 -- 172.21.15.7:0/2825772968 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27680b59a0 msgr2=0x7f27680ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2766ffd640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27680b59a0 0x7f27680ae330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2766ffd640 1 -- 172.21.15.7:0/2825772968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27681499e0 con 0x7f2768072440 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f2766ffd640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2768072440 0x7f27680addf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f275000d800 tx=0x7f275000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f275ffff640 1 -- 172.21.15.7:0/2825772968 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275000b6c0 con 0x7f2768072440 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f275ffff640 1 -- 172.21.15.7:0/2825772968 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2750019070 con 0x7f2768072440 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f275ffff640 1 -- 172.21.15.7:0/2825772968 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275000baa0 con 0x7f2768072440 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276814a770 con 0x7f2768072440 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.727+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27680b1c70 con 0x7f2768072440 2024-03-19T01:23:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.731+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2768075570 con 0x7f2768072440 2024-03-19T01:23:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.731+0000 7f275ffff640 1 -- 172.21.15.7:0/2825772968 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f275000bc40 con 0x7f2768072440 2024-03-19T01:23:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.731+0000 7f275ffff640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2748067660 0x7f2748069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.731+0000 7f275ffff640 1 -- 172.21.15.7:0/2825772968 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2750085d30 con 0x7f2768072440 2024-03-19T01:23:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.731+0000 7f27677fe640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2748067660 0x7f2748069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:49.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.731+0000 7f27677fe640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2748067660 0x7f2748069b20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f27580ad000 tx=0x7f2758008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:49.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.735+0000 7f275ffff640 1 -- 172.21.15.7:0/2825772968 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f27500569b0 con 0x7f2768072440 2024-03-19T01:23:49.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2768142af0 con 0x7f2768072440 2024-03-19T01:23:49.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f275ffff640 1 -- 172.21.15.7:0/2825772968 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f27500562b0 con 0x7f2768072440 2024-03-19T01:23:49.896 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2748067660 msgr2=0x7f2748069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2748067660 0x7f2748069b20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f27580ad000 tx=0x7f2758008040 comp rx=0 tx=0).stop 2024-03-19T01:23:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2768072440 msgr2=0x7f27680addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2768072440 0x7f27680addf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f275000d800 tx=0x7f275000dcc0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 shutdown_connections 2024-03-19T01:23:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2748067660 0x7f2748069b20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27680b59a0 0x7f27680ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2768072440 0x7f27680addf0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 --2- 172.21.15.7:0/2825772968 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f276806ff90 0x7f27681e3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 >> 172.21.15.7:0/2825772968 conn(0x7f276806c3b0 msgr2=0x7f27680b7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 shutdown_connections 2024-03-19T01:23:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:49.895+0000 7f276e16d640 1 -- 172.21.15.7:0/2825772968 wait complete. 2024-03-19T01:23:49.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:50.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f4590c0d640 1 Processor -- start 2024-03-19T01:23:50.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f4590c0d640 1 -- start start 2024-03-19T01:23:50.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f4590c0d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f458c0b5780 0x7f458c14aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f4590c0d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f458c14fd80 0x7f458c152170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f4590c0d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f458c0afee0 con 0x7f458c149e50 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f4590c0d640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f458c0b6250 con 0x7f458c0b5780 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f4590c0d640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f458c0afa50 con 0x7f458c14fd80 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f458a59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f458c0b5780 0x7f458c14aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f4589d9b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f458c149e50 0x7f458c14a2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35092/0 (socket says 172.21.15.7:35092) 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f458a59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f458c0b5780 0x7f458c14aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42990/0 (socket says 172.21.15.7:42990) 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f458a59c640 1 -- 172.21.15.7:0/2980869730 learned_addr learned my addr 172.21.15.7:0/2980869730 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f458ad9d640 1 --2- 172.21.15.7:0/2980869730 >> v2:172.21.15.167:3300/0 conn(0x7f458c14fd80 0x7f458c152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f458a59c640 1 -- 172.21.15.7:0/2980869730 >> v2:172.21.15.167:3300/0 conn(0x7f458c14fd80 msgr2=0x7f458c152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:50.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f458a59c640 1 --2- 172.21.15.7:0/2980869730 >> v2:172.21.15.167:3300/0 conn(0x7f458c14fd80 0x7f458c152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f458a59c640 1 -- 172.21.15.7:0/2980869730 >> v1:172.21.15.7:6789/0 conn(0x7f458c149e50 legacy=0x7f458c14a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f458959a640 1 -- 172.21.15.7:0/2980869730 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573756119 0 0) 0x7f458c0afee0 con 0x7f458c149e50 2024-03-19T01:23:50.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.059+0000 7f458a59c640 1 -- 172.21.15.7:0/2980869730 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c0b5b60 con 0x7f458c0b5780 2024-03-19T01:23:50.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458a59c640 1 --2- 172.21.15.7:0/2980869730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c0b5780 0x7f458c14aa60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f458c0b0b70 tx=0x7f457c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd3764574a672e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:50.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458959a640 1 -- 172.21.15.7:0/2980869730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c0a5620 con 0x7f458c0b5780 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458959a640 1 -- 172.21.15.7:0/2980869730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f457c0a57c0 con 0x7f458c0b5780 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458959a640 1 -- 172.21.15.7:0/2980869730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c0a5b90 con 0x7f458c0b5780 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 -- 172.21.15.7:0/2980869730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c0b5780 msgr2=0x7f458c14aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 --2- 172.21.15.7:0/2980869730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c0b5780 0x7f458c14aa60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f458c0b0b70 tx=0x7f457c01c960 comp rx=0 tx=0).stop 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 -- 172.21.15.7:0/2980869730 shutdown_connections 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 --2- 172.21.15.7:0/2980869730 >> v2:172.21.15.167:3300/0 conn(0x7f458c14fd80 0x7f458c152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 --2- 172.21.15.7:0/2980869730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c0b5780 0x7f458c14aa60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 -- 172.21.15.7:0/2980869730 >> 172.21.15.7:0/2980869730 conn(0x7f458c0b6610 msgr2=0x7f458c0b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 -- 172.21.15.7:0/2980869730 shutdown_connections 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 -- 172.21.15.7:0/2980869730 wait complete. 2024-03-19T01:23:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 Processor -- start 2024-03-19T01:23:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 -- start start 2024-03-19T01:23:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f458c0b5780 0x7f458c14eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f458c149e50 0x7f458c14c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c14fd80 0x7f458c14dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f458c06ec60 con 0x7f458c14fd80 2024-03-19T01:23:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f458c072d70 con 0x7f458c149e50 2024-03-19T01:23:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4590c0d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f458c1486b0 con 0x7f458c0b5780 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c14fd80 0x7f458c14dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c14fd80 0x7f458c14dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43006/0 (socket says 172.21.15.7:43006) 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458ad9d640 1 -- 172.21.15.7:0/433831395 learned_addr learned my addr 172.21.15.7:0/433831395 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f4589d9b640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f458c149e50 0x7f458c14c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458a59c640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f458c0b5780 0x7f458c14eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458ad9d640 1 -- 172.21.15.7:0/433831395 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f458c0b5780 msgr2=0x7f458c14eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458ad9d640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f458c0b5780 0x7f458c14eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458ad9d640 1 -- 172.21.15.7:0/433831395 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f458c149e50 msgr2=0x7f458c14c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458ad9d640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f458c149e50 0x7f458c14c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.063+0000 7f458ad9d640 1 -- 172.21.15.7:0/433831395 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c14f1e0 con 0x7f458c14fd80 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f458ad9d640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c14fd80 0x7f458c14dcb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f458400d560 tx=0x7f458400da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f4582ffd640 1 -- 172.21.15.7:0/433831395 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45840125f0 con 0x7f458c14fd80 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f4582ffd640 1 -- 172.21.15.7:0/433831395 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4584019070 con 0x7f458c14fd80 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f458c14f4c0 con 0x7f458c14fd80 2024-03-19T01:23:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f4582ffd640 1 -- 172.21.15.7:0/433831395 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45840129f0 con 0x7f458c14fd80 2024-03-19T01:23:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f458c0b23e0 con 0x7f458c14fd80 2024-03-19T01:23:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f458c075570 con 0x7f458c14fd80 2024-03-19T01:23:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f4582ffd640 1 -- 172.21.15.7:0/433831395 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4584012b90 con 0x7f458c14fd80 2024-03-19T01:23:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f4582ffd640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f456c067660 0x7f456c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f4582ffd640 1 -- 172.21.15.7:0/433831395 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4584085af0 con 0x7f458c14fd80 2024-03-19T01:23:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f458a59c640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f456c067660 0x7f456c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.067+0000 7f458a59c640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f456c067660 0x7f456c069b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f457c0ae000 tx=0x7f457c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:50.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.071+0000 7f4582ffd640 1 -- 172.21.15.7:0/433831395 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f45840561b0 con 0x7f458c14fd80 2024-03-19T01:23:50.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.231+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f458c06ede0 con 0x7f458c14fd80 2024-03-19T01:23:50.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.231+0000 7f4582ffd640 1 -- 172.21.15.7:0/433831395 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f458405a250 con 0x7f458c14fd80 2024-03-19T01:23:50.233 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:50.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f456c067660 msgr2=0x7f456c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f456c067660 0x7f456c069b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f457c0ae000 tx=0x7f457c0a7070 comp rx=0 tx=0).stop 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c14fd80 msgr2=0x7f458c14dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c14fd80 0x7f458c14dcb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f458400d560 tx=0x7f458400da20 comp rx=0 tx=0).stop 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 shutdown_connections 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f456c067660 0x7f456c069b20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f458c14fd80 0x7f458c14dcb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f458c149e50 0x7f458c14c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 --2- 172.21.15.7:0/433831395 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f458c0b5780 0x7f458c14eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 >> 172.21.15.7:0/433831395 conn(0x7f458c0b6610 msgr2=0x7f458c149290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 shutdown_connections 2024-03-19T01:23:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.235+0000 7f4590c0d640 1 -- 172.21.15.7:0/433831395 wait complete. 2024-03-19T01:23:50.299 INFO:tasks.cephfs_test_runner:Starting test: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) Test that when cd is issued with an argument, CWD is changed 2024-03-19T01:23:50.300 INFO:tasks.cephfs_test_runner:test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-19T01:23:50.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:50.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde0b104640 1 Processor -- start 2024-03-19T01:23:50.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde0b104640 1 -- start start 2024-03-19T01:23:50.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde0b104640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde040b3a20 0x7fde04150c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde0b104640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde04072a90 con 0x7fde040726b0 2024-03-19T01:23:50.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde0b104640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde040b4a30 con 0x7fde040ba710 2024-03-19T01:23:50.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde0b104640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fde040b4750 con 0x7fde040b3a20 2024-03-19T01:23:50.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde096a1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde040b3a20 0x7fde04150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde08ea0640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fde040726b0 0x7fde04148940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35108/0 (socket says 172.21.15.7:35108) 2024-03-19T01:23:50.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde08ea0640 1 -- 172.21.15.7:0/2443094984 learned_addr learned my addr 172.21.15.7:0/2443094984 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:50.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde096a1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde040b3a20 0x7fde04150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43016/0 (socket says 172.21.15.7:43016) 2024-03-19T01:23:50.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde096a1640 1 -- 172.21.15.7:0/2443094984 >> v1:172.21.15.167:6789/0 conn(0x7fde040ba710 legacy=0x7fde040b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:50.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde096a1640 1 -- 172.21.15.7:0/2443094984 >> v1:172.21.15.7:6789/0 conn(0x7fde040726b0 legacy=0x7fde04148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.407+0000 7fde096a1640 1 -- 172.21.15.7:0/2443094984 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde040b5470 con 0x7fde040b3a20 2024-03-19T01:23:50.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde096a1640 1 --2- 172.21.15.7:0/2443094984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040b3a20 0x7fde04150c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009d90 tx=0x7fddfc01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184cfb55eade0370 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:50.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde037fe640 1 -- 172.21.15.7:0/2443094984 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0b2070 con 0x7fde040b3a20 2024-03-19T01:23:50.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde037fe640 1 -- 172.21.15.7:0/2443094984 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddfc0a58f0 con 0x7fde040b3a20 2024-03-19T01:23:50.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde037fe640 1 -- 172.21.15.7:0/2443094984 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0a5cc0 con 0x7fde040b3a20 2024-03-19T01:23:50.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde0b104640 1 -- 172.21.15.7:0/2443094984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040b3a20 msgr2=0x7fde04150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde0b104640 1 --2- 172.21.15.7:0/2443094984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040b3a20 0x7fde04150c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009d90 tx=0x7fddfc01c870 comp rx=0 tx=0).stop 2024-03-19T01:23:50.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde0b104640 1 -- 172.21.15.7:0/2443094984 shutdown_connections 2024-03-19T01:23:50.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde0b104640 1 --2- 172.21.15.7:0/2443094984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040b3a20 0x7fde04150c80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde0b104640 1 -- 172.21.15.7:0/2443094984 >> 172.21.15.7:0/2443094984 conn(0x7fde04074130 msgr2=0x7fde04074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:50.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde0b104640 1 -- 172.21.15.7:0/2443094984 shutdown_connections 2024-03-19T01:23:50.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.411+0000 7fde0b104640 1 -- 172.21.15.7:0/2443094984 wait complete. 2024-03-19T01:23:50.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 Processor -- start 2024-03-19T01:23:50.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 -- start start 2024-03-19T01:23:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040726b0 0x7fde04071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde040b3a20 0x7fde041e0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde040ba710 0x7fde041e09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fde04153b60 con 0x7fde040726b0 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fde041539e0 con 0x7fde040b3a20 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fde04153ce0 con 0x7fde040ba710 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde08ea0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040726b0 0x7fde04071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde08ea0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040726b0 0x7fde04071ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43018/0 (socket says 172.21.15.7:43018) 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde08ea0640 1 -- 172.21.15.7:0/3368336288 learned_addr learned my addr 172.21.15.7:0/3368336288 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde096a1640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde040ba710 0x7fde041e09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde03fff640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde040b3a20 0x7fde041e0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde08ea0640 1 -- 172.21.15.7:0/3368336288 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde040ba710 msgr2=0x7fde041e09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde08ea0640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde040ba710 0x7fde041e09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde08ea0640 1 -- 172.21.15.7:0/3368336288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde040b3a20 msgr2=0x7fde041e0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde08ea0640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde040b3a20 0x7fde041e0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde08ea0640 1 -- 172.21.15.7:0/3368336288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde041dffd0 con 0x7fde040726b0 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde08ea0640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040726b0 0x7fde04071ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fddf800c420 tx=0x7fddf800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde017fa640 1 -- 172.21.15.7:0/3368336288 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8017070 con 0x7fde040726b0 2024-03-19T01:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde017fa640 1 -- 172.21.15.7:0/3368336288 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddf8005cf0 con 0x7fde040726b0 2024-03-19T01:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde040ae3b0 con 0x7fde040726b0 2024-03-19T01:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde017fa640 1 -- 172.21.15.7:0/3368336288 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf80072f0 con 0x7fde040726b0 2024-03-19T01:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde04070860 con 0x7fde040726b0 2024-03-19T01:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.415+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde04075460 con 0x7fde040726b0 2024-03-19T01:23:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.419+0000 7fde017fa640 1 -- 172.21.15.7:0/3368336288 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fddf801a070 con 0x7fde040726b0 2024-03-19T01:23:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.419+0000 7fde017fa640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdde8067660 0x7fdde8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.419+0000 7fde03fff640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdde8067660 0x7fdde8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.419+0000 7fde017fa640 1 -- 172.21.15.7:0/3368336288 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fddf8084c00 con 0x7fde040726b0 2024-03-19T01:23:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.419+0000 7fde03fff640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdde8067660 0x7fdde8069b20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde041432f0 tx=0x7fddf0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.427+0000 7fde017fa640 1 -- 172.21.15.7:0/3368336288 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fddf808b080 con 0x7fde040726b0 2024-03-19T01:23:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.619+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde04071560 con 0x7fde040726b0 2024-03-19T01:23:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.619+0000 7fde017fa640 1 -- 172.21.15.7:0/3368336288 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7fde041dffd0 con 0x7fde040726b0 2024-03-19T01:23:50.621 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdde8067660 msgr2=0x7fdde8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdde8067660 0x7fdde8069b20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde041432f0 tx=0x7fddf0008040 comp rx=0 tx=0).stop 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040726b0 msgr2=0x7fde04071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040726b0 0x7fde04071ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fddf800c420 tx=0x7fddf800c8e0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 shutdown_connections 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdde8067660 0x7fdde8069b20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde040ba710 0x7fde041e09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde040b3a20 0x7fde041e0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 --2- 172.21.15.7:0/3368336288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde040726b0 0x7fde04071ab0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 >> 172.21.15.7:0/3368336288 conn(0x7fde04074130 msgr2=0x7fde04067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 shutdown_connections 2024-03-19T01:23:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.623+0000 7fde0b104640 1 -- 172.21.15.7:0/3368336288 wait complete. 2024-03-19T01:23:50.639 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:50.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:50.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f18208f7640 1 Processor -- start 2024-03-19T01:23:50.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f18208f7640 1 -- start start 2024-03-19T01:23:50.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f18208f7640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f181c0b56e0 0x7f181c0b4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f18208f7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f181c0b48c0 0x7f181c14fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f18208f7640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f181c0b2c20 con 0x7f181c0b77f0 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f18208f7640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f181c0b5d00 con 0x7f181c0b48c0 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f18208f7640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f181c0b5fe0 con 0x7f181c0b56e0 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f181ad9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f181c0b48c0 0x7f181c14fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f181ad9d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f181c0b48c0 0x7f181c14fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43032/0 (socket says 172.21.15.7:43032) 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f181ad9d640 1 -- 172.21.15.7:0/3189106016 learned_addr learned my addr 172.21.15.7:0/3189106016 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f181ad9d640 1 -- 172.21.15.7:0/3189106016 >> v2:172.21.15.167:3300/0 conn(0x7f181c0b56e0 msgr2=0x7f181c0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f1813fff640 1 --2- 172.21.15.7:0/3189106016 >> v2:172.21.15.167:3300/0 conn(0x7f181c0b56e0 0x7f181c0b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f181ad9d640 1 --2- 172.21.15.7:0/3189106016 >> v2:172.21.15.167:3300/0 conn(0x7f181c0b56e0 0x7f181c0b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f181ad9d640 1 -- 172.21.15.7:0/3189106016 >> v1:172.21.15.167:6789/0 conn(0x7f181c0b77f0 legacy=0x7f181c0b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f181ad9d640 1 -- 172.21.15.7:0/3189106016 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181c0b64d0 con 0x7f181c0b48c0 2024-03-19T01:23:50.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f181ad9d640 1 --2- 172.21.15.7:0/3189106016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 0x7f181c14fb60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f181c072b20 tx=0x7f181401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506a22f48e929142 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:50.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f1819d9b640 1 -- 172.21.15.7:0/3189106016 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18140b0070 con 0x7f181c0b48c0 2024-03-19T01:23:50.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f1819d9b640 1 -- 172.21.15.7:0/3189106016 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18140ab600 con 0x7f181c0b48c0 2024-03-19T01:23:50.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.775+0000 7f1819d9b640 1 -- 172.21.15.7:0/3189106016 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18140ab9d0 con 0x7f181c0b48c0 2024-03-19T01:23:50.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- 172.21.15.7:0/3189106016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 msgr2=0x7f181c14fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 --2- 172.21.15.7:0/3189106016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 0x7f181c14fb60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f181c072b20 tx=0x7f181401c490 comp rx=0 tx=0).stop 2024-03-19T01:23:50.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- 172.21.15.7:0/3189106016 shutdown_connections 2024-03-19T01:23:50.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 --2- 172.21.15.7:0/3189106016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 0x7f181c14fb60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 --2- 172.21.15.7:0/3189106016 >> v2:172.21.15.167:3300/0 conn(0x7f181c0b56e0 0x7f181c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- 172.21.15.7:0/3189106016 >> 172.21.15.7:0/3189106016 conn(0x7f181c06b2c0 msgr2=0x7f181c06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- 172.21.15.7:0/3189106016 shutdown_connections 2024-03-19T01:23:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- 172.21.15.7:0/3189106016 wait complete. 2024-03-19T01:23:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 Processor -- start 2024-03-19T01:23:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- start start 2024-03-19T01:23:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 0x7f181c0ba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f181c0b56e0 0x7f181c1381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181c0b77f0 0x7f181c138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f181c1489b0 con 0x7f181c0b48c0 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f181c148830 con 0x7f181c0b56e0 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f181c1528c0 con 0x7f181c0b77f0 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 0x7f181c0ba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 0x7f181c0ba1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43044/0 (socket says 172.21.15.7:43044) 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181a59c640 1 -- 172.21.15.7:0/3627005858 learned_addr learned my addr 172.21.15.7:0/3627005858 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181ad9d640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181c0b77f0 0x7f181c138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f1813fff640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f181c0b56e0 0x7f181c1381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181a59c640 1 -- 172.21.15.7:0/3627005858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181c0b77f0 msgr2=0x7f181c138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181a59c640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181c0b77f0 0x7f181c138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181a59c640 1 -- 172.21.15.7:0/3627005858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f181c0b56e0 msgr2=0x7f181c1381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181a59c640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f181c0b56e0 0x7f181c1381d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181a59c640 1 -- 172.21.15.7:0/3627005858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181c1399b0 con 0x7f181c0b48c0 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f1813fff640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f181c0b56e0 0x7f181c1381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f181a59c640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 0x7f181c0ba1d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f180c00c730 tx=0x7f180c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f1812ffd640 1 -- 172.21.15.7:0/3627005858 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c016020 con 0x7f181c0b48c0 2024-03-19T01:23:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f1812ffd640 1 -- 172.21.15.7:0/3627005858 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f180c006d60 con 0x7f181c0b48c0 2024-03-19T01:23:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f181c136f60 con 0x7f181c0b48c0 2024-03-19T01:23:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f1812ffd640 1 -- 172.21.15.7:0/3627005858 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c003bc0 con 0x7f181c0b48c0 2024-03-19T01:23:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f181c137560 con 0x7f181c0b48c0 2024-03-19T01:23:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.779+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f181c075570 con 0x7f181c0b48c0 2024-03-19T01:23:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.783+0000 7f1812ffd640 1 -- 172.21.15.7:0/3627005858 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f180c00a850 con 0x7f181c0b48c0 2024-03-19T01:23:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.783+0000 7f1812ffd640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f17fc067660 0x7f17fc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.783+0000 7f1812ffd640 1 -- 172.21.15.7:0/3627005858 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f180c085590 con 0x7f181c0b48c0 2024-03-19T01:23:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.783+0000 7f1813fff640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f17fc067660 0x7f17fc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.783+0000 7f1813fff640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f17fc067660 0x7f17fc069b20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1804006fd0 tx=0x7f1804008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.787+0000 7f1812ffd640 1 -- 172.21.15.7:0/3627005858 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f180c020020 con 0x7f181c0b48c0 2024-03-19T01:23:50.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.943+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f181c137280 con 0x7f181c0b48c0 2024-03-19T01:23:50.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.943+0000 7f1812ffd640 1 -- 172.21.15.7:0/3627005858 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f180c056220 con 0x7f181c0b48c0 2024-03-19T01:23:50.945 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:50.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f17fc067660 msgr2=0x7f17fc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f17fc067660 0x7f17fc069b20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1804006fd0 tx=0x7f1804008040 comp rx=0 tx=0).stop 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 msgr2=0x7f181c0ba1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 0x7f181c0ba1d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f180c00c730 tx=0x7f180c00cbf0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 shutdown_connections 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f17fc067660 0x7f17fc069b20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181c0b77f0 0x7f181c138710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f181c0b56e0 0x7f181c1381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 --2- 172.21.15.7:0/3627005858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f181c0b48c0 0x7f181c0ba1d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 >> 172.21.15.7:0/3627005858 conn(0x7f181c06b2c0 msgr2=0x7f181c143b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 shutdown_connections 2024-03-19T01:23:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:50.947+0000 7f18208f7640 1 -- 172.21.15.7:0/3627005858 wait complete. 2024-03-19T01:23:50.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167cb2e640 1 Processor -- start 2024-03-19T01:23:51.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167cb2e640 1 -- start start 2024-03-19T01:23:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167cb2e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16780714a0 0x7f1678072110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167cb2e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 0x7f167813e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167cb2e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16780b0c70 con 0x7f16781443f0 2024-03-19T01:23:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167cb2e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f1678071880 con 0x7f16780714a0 2024-03-19T01:23:51.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167cb2e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1678072770 con 0x7f167813e030 2024-03-19T01:23:51.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16780714a0 0x7f1678072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f1676d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f16781443f0 0x7f1678144880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35118/0 (socket says 172.21.15.7:35118) 2024-03-19T01:23:51.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16780714a0 0x7f1678072110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43058/0 (socket says 172.21.15.7:43058) 2024-03-19T01:23:51.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f1676d9d640 1 -- 172.21.15.7:0/2375988378 learned_addr learned my addr 172.21.15.7:0/2375988378 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:51.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f1675d9b640 1 --2- 172.21.15.7:0/2375988378 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 0x7f167813e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167659c640 1 -- 172.21.15.7:0/2375988378 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 msgr2=0x7f167813e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:51.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167559a640 1 -- 172.21.15.7:0/2375988378 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 79510159 0 0) 0x7f16780b0c70 con 0x7f16781443f0 2024-03-19T01:23:51.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167659c640 1 --2- 172.21.15.7:0/2375988378 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 0x7f167813e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167659c640 1 -- 172.21.15.7:0/2375988378 >> v1:172.21.15.7:6789/0 conn(0x7f16781443f0 legacy=0x7f1678144880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.087+0000 7f167659c640 1 -- 172.21.15.7:0/2375988378 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1678070a60 con 0x7f16780714a0 2024-03-19T01:23:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167659c640 1 --2- 172.21.15.7:0/2375988378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 0x7f1678072110 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f16680097c0 tx=0x7f166801c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb989a31511e8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167559a640 1 -- 172.21.15.7:0/2375988378 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680b0070 con 0x7f16780714a0 2024-03-19T01:23:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167559a640 1 -- 172.21.15.7:0/2375988378 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1668004230 con 0x7f16780714a0 2024-03-19T01:23:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167559a640 1 -- 172.21.15.7:0/2375988378 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668003620 con 0x7f16780714a0 2024-03-19T01:23:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167cb2e640 1 -- 172.21.15.7:0/2375988378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 msgr2=0x7f1678072110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167cb2e640 1 --2- 172.21.15.7:0/2375988378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 0x7f1678072110 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f16680097c0 tx=0x7f166801c350 comp rx=0 tx=0).stop 2024-03-19T01:23:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167cb2e640 1 -- 172.21.15.7:0/2375988378 shutdown_connections 2024-03-19T01:23:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167cb2e640 1 --2- 172.21.15.7:0/2375988378 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 0x7f167813e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167cb2e640 1 --2- 172.21.15.7:0/2375988378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 0x7f1678072110 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167cb2e640 1 -- 172.21.15.7:0/2375988378 >> 172.21.15.7:0/2375988378 conn(0x7f16780b7980 msgr2=0x7f16780b7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167cb2e640 1 -- 172.21.15.7:0/2375988378 shutdown_connections 2024-03-19T01:23:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.091+0000 7f167cb2e640 1 -- 172.21.15.7:0/2375988378 wait complete. 2024-03-19T01:23:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 Processor -- start 2024-03-19T01:23:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 -- start start 2024-03-19T01:23:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 0x7f16780b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 0x7f16781e11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f16781443f0 0x7f16781dd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1678151380 con 0x7f16780714a0 2024-03-19T01:23:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1678151200 con 0x7f16781443f0 2024-03-19T01:23:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1678151500 con 0x7f167813e030 2024-03-19T01:23:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 0x7f16780b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 0x7f16780b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43062/0 (socket says 172.21.15.7:43062) 2024-03-19T01:23:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167659c640 1 -- 172.21.15.7:0/3438351416 learned_addr learned my addr 172.21.15.7:0/3438351416 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f1675d9b640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 0x7f16781e11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f1676d9d640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f16781443f0 0x7f16781dd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167659c640 1 -- 172.21.15.7:0/3438351416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 msgr2=0x7f16781e11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167659c640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 0x7f16781e11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167659c640 1 -- 172.21.15.7:0/3438351416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f16781443f0 msgr2=0x7f16781dd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167659c640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f16781443f0 0x7f16781dd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167659c640 1 -- 172.21.15.7:0/3438351416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16780ab7f0 con 0x7f16780714a0 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f1676d9d640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f16781443f0 0x7f16781dd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167659c640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 0x7f16780b3250 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1668019f80 tx=0x7f16680190f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f1666ffd640 1 -- 172.21.15.7:0/3438351416 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680b9070 con 0x7f16780714a0 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f1666ffd640 1 -- 172.21.15.7:0/3438351416 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16680043e0 con 0x7f16780714a0 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167814e780 con 0x7f16780714a0 2024-03-19T01:23:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f1666ffd640 1 -- 172.21.15.7:0/3438351416 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680b0040 con 0x7f16780714a0 2024-03-19T01:23:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16780b3c30 con 0x7f16780714a0 2024-03-19T01:23:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1678075570 con 0x7f16780714a0 2024-03-19T01:23:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.095+0000 7f1666ffd640 1 -- 172.21.15.7:0/3438351416 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f16680bc070 con 0x7f16780714a0 2024-03-19T01:23:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.099+0000 7f1666ffd640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1658067660 0x7f1658069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.099+0000 7f1675d9b640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1658067660 0x7f1658069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.099+0000 7f1666ffd640 1 -- 172.21.15.7:0/3438351416 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f166811fa00 con 0x7f16780714a0 2024-03-19T01:23:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.099+0000 7f1675d9b640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1658067660 0x7f1658069b20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f16780b0650 tx=0x7f166c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:51.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.103+0000 7f1666ffd640 1 -- 172.21.15.7:0/3438351416 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1668016090 con 0x7f16780714a0 2024-03-19T01:23:51.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1678141d70 con 0x7f16780714a0 2024-03-19T01:23:51.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f1666ffd640 1 -- 172.21.15.7:0/3438351416 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f16680f0830 con 0x7f16780714a0 2024-03-19T01:23:51.263 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:51.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1658067660 msgr2=0x7f1658069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1658067660 0x7f1658069b20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f16780b0650 tx=0x7f166c008040 comp rx=0 tx=0).stop 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 msgr2=0x7f16780b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 0x7f16780b3250 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1668019f80 tx=0x7f16680190f0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 shutdown_connections 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1658067660 0x7f1658069b20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167813e030 0x7f16781e11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f16781443f0 0x7f16781dd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 --2- 172.21.15.7:0/3438351416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16780714a0 0x7f16780b3250 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 >> 172.21.15.7:0/3438351416 conn(0x7f16780b7980 msgr2=0x7f167813c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 shutdown_connections 2024-03-19T01:23:51.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.263+0000 7f167cb2e640 1 -- 172.21.15.7:0/3438351416 wait complete. 2024-03-19T01:23:51.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.399+0000 7f627056f640 1 Processor -- start 2024-03-19T01:23:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.399+0000 7f627056f640 1 -- start start 2024-03-19T01:23:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f627056f640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f62680b1c50 0x7f62680afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f627056f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62680b2170 con 0x7f62680b0ce0 2024-03-19T01:23:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f627056f640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62680b2450 con 0x7f62680b0900 2024-03-19T01:23:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f627056f640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f6268073830 con 0x7f62680b1c50 2024-03-19T01:23:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626db0a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f62680b0ce0 0x7f626814c1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35120/0 (socket says 172.21.15.7:35120) 2024-03-19T01:23:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626db0a640 1 -- 172.21.15.7:0/1305005917 learned_addr learned my addr 172.21.15.7:0/1305005917 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626eb0c640 1 --2- 172.21.15.7:0/1305005917 >> v2:172.21.15.167:3300/0 conn(0x7f62680b1c50 0x7f62680afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 264234627 0 0) 0x7f62680b2170 con 0x7f62680b0ce0 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6254003530 con 0x7f62680b0ce0 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3646157449 0 0) 0x7f6254003530 con 0x7f62680b0ce0 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 >> v2:172.21.15.167:3300/0 conn(0x7f62680b1c50 msgr2=0x7f62680afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 --2- 172.21.15.7:0/1305005917 >> v2:172.21.15.167:3300/0 conn(0x7f62680b1c50 0x7f62680afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 >> v1:172.21.15.167:6789/0 conn(0x7f62680b0900 legacy=0x7f6268073120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62680b1160 con 0x7f62680b0ce0 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f6258001680 con 0x7f62680b0ce0 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 >> v1:172.21.15.7:6789/0 conn(0x7f62680b0ce0 legacy=0x7f626814c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f6254003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6254004810 0x7f6254004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540052f0 0x7f625400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6268073830 con 0x7f62540039c0 2024-03-19T01:23:51.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f625400c1c0 con 0x7f62540052f0 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f625400c3c0 con 0x7f6254004810 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626e30b640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f6254003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626db0a640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6254004810 0x7f6254004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626eb0c640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540052f0 0x7f625400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626e30b640 1 -- 172.21.15.7:0/1305005917 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6254004810 msgr2=0x7f6254004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626e30b640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6254004810 0x7f6254004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626e30b640 1 -- 172.21.15.7:0/1305005917 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540052f0 msgr2=0x7f625400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626e30b640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540052f0 0x7f625400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626e30b640 1 -- 172.21.15.7:0/1305005917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6254003530 con 0x7f62540039c0 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.403+0000 7f626eb0c640 1 -- 172.21.15.7:0/1305005917 reap_dead start 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626e30b640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f6254003de0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6264001750 tx=0x7f626401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2885265f59206152 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62640013a0 con 0x7f62540039c0 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62640052f0 con 0x7f62540039c0 2024-03-19T01:23:51.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626d309640 1 -- 172.21.15.7:0/1305005917 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6264003840 con 0x7f62540039c0 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 -- 172.21.15.7:0/1305005917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 msgr2=0x7f6254003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f6254003de0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6264001750 tx=0x7f626401c3c0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 -- 172.21.15.7:0/1305005917 shutdown_connections 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 --2- 172.21.15.7:0/1305005917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f6254003de0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 -- 172.21.15.7:0/1305005917 >> 172.21.15.7:0/1305005917 conn(0x7f62680b9120 msgr2=0x7f62680b9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 -- 172.21.15.7:0/1305005917 shutdown_connections 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 -- 172.21.15.7:0/1305005917 wait complete. 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 Processor -- start 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 -- start start 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f626813c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6268141210 0x7f62681d2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62681e1e80 0x7f62681e2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f62680734b0 con 0x7f62540039c0 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6268073330 con 0x7f62681e1e80 2024-03-19T01:23:51.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f627056f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6268073630 con 0x7f6268141210 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626e30b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f626813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626e30b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f626813c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43076/0 (socket says 172.21.15.7:43076) 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626e30b640 1 -- 172.21.15.7:0/1343475366 learned_addr learned my addr 172.21.15.7:0/1343475366 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626db0a640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6268141210 0x7f62681d2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626eb0c640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62681e1e80 0x7f62681e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626e30b640 1 -- 172.21.15.7:0/1343475366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6268141210 msgr2=0x7f62681d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626e30b640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6268141210 0x7f62681d2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626e30b640 1 -- 172.21.15.7:0/1343475366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62681e1e80 msgr2=0x7f62681e2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626e30b640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62681e1e80 0x7f62681e2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.407+0000 7f626e30b640 1 -- 172.21.15.7:0/1343475366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62681e2870 con 0x7f62540039c0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f626e30b640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f626813c380 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6264005040 tx=0x7f6254005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f625effd640 1 -- 172.21.15.7:0/1343475366 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62640b8020 con 0x7f62540039c0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f625effd640 1 -- 172.21.15.7:0/1343475366 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62640054a0 con 0x7f62540039c0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f625effd640 1 -- 172.21.15.7:0/1343475366 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6264007a40 con 0x7f62540039c0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626813f140 con 0x7f62540039c0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62681d25a0 con 0x7f62540039c0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6268075570 con 0x7f62540039c0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f625effd640 1 -- 172.21.15.7:0/1343475366 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f62640bb020 con 0x7f62540039c0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f625effd640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f624c067690 0x7f624c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f625effd640 1 -- 172.21.15.7:0/1343475366 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f626411f9f0 con 0x7f62540039c0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f626db0a640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f624c067690 0x7f624c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.411+0000 7f626db0a640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f624c067690 0x7f624c069b50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f62680b8fa0 tx=0x7f6258009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:51.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.415+0000 7f625effd640 1 -- 172.21.15.7:0/1343475366 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f62640c2030 con 0x7f62540039c0 2024-03-19T01:23:51.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.579+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f626813ee60 con 0x7f62540039c0 2024-03-19T01:23:51.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.579+0000 7f625effd640 1 -- 172.21.15.7:0/1343475366 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f62640160d0 con 0x7f62540039c0 2024-03-19T01:23:51.583 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:51.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f624c067690 msgr2=0x7f624c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f624c067690 0x7f624c069b50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f62680b8fa0 tx=0x7f6258009040 comp rx=0 tx=0).stop 2024-03-19T01:23:51.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 msgr2=0x7f626813c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f626813c380 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6264005040 tx=0x7f6254005840 comp rx=0 tx=0).stop 2024-03-19T01:23:51.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 shutdown_connections 2024-03-19T01:23:51.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f624c067690 0x7f624c069b50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6268141210 0x7f62681d2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62681e1e80 0x7f62681e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 --2- 172.21.15.7:0/1343475366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62540039c0 0x7f626813c380 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 >> 172.21.15.7:0/1343475366 conn(0x7f62680b9120 msgr2=0x7f626813d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:51.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 shutdown_connections 2024-03-19T01:23:51.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.583+0000 7f627056f640 1 -- 172.21.15.7:0/1343475366 wait complete. 2024-03-19T01:23:51.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args' 2024-03-19T01:23:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4f48d640 1 Processor -- start 2024-03-19T01:23:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4f48d640 1 -- start start 2024-03-19T01:23:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4f48d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fdd48072bf0 0x7fdd48147850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4f48d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd48147d90 0x7fdd48148210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4f48d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd481488d0 con 0x7fdd480b56e0 2024-03-19T01:23:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4f48d640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fdd4806ee70 con 0x7fdd48072bf0 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4f48d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdd480b5ac0 con 0x7fdd48147d90 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4d229640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdd480b56e0 0x7fdd4806d5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35128/0 (socket says 172.21.15.7:35128) 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4d229640 1 -- 172.21.15.7:0/4141575097 learned_addr learned my addr 172.21.15.7:0/4141575097 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4da2a640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd48147d90 0x7fdd48148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4ca28640 1 --2- 172.21.15.7:0/4141575097 >> v2:172.21.15.167:3300/0 conn(0x7fdd48072bf0 0x7fdd48147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092398891 0 0) 0x7fdd481488d0 con 0x7fdd480b56e0 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd34003530 con 0x7fdd480b56e0 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1485075304 0 0) 0x7fdd34003530 con 0x7fdd480b56e0 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd48147d90 msgr2=0x7fdd48148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd48147d90 0x7fdd48148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 >> v2:172.21.15.167:3300/0 conn(0x7fdd48072bf0 msgr2=0x7fdd48147850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 --2- 172.21.15.7:0/4141575097 >> v2:172.21.15.167:3300/0 conn(0x7fdd48072bf0 0x7fdd48147850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd48071120 con 0x7fdd480b56e0 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fdd44001680 con 0x7fdd480b56e0 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 >> v1:172.21.15.7:6789/0 conn(0x7fdd480b56e0 legacy=0x7fdd4806d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd340039c0 0x7fdd34003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd48147d90 0x7fdd34004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd34004d50 0x7fdd340051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4806ee70 con 0x7fdd340039c0 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdd480b5ac0 con 0x7fdd34004d50 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3400ba80 con 0x7fdd48147d90 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4d229640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd340039c0 0x7fdd34003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4ca28640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd48147d90 0x7fdd34004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4da2a640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd34004d50 0x7fdd340051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4d229640 1 -- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd48147d90 msgr2=0x7fdd34004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4d229640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd48147d90 0x7fdd34004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4d229640 1 -- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd34004d50 msgr2=0x7fdd340051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:51.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4d229640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd34004d50 0x7fdd340051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.719+0000 7fdd4d229640 1 -- 172.21.15.7:0/4141575097 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd34003530 con 0x7fdd340039c0 2024-03-19T01:23:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4d229640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd340039c0 0x7fdd34003e00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c290 tx=0x7fdd4401cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1941eff18ad84bde server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44007d60 con 0x7fdd340039c0 2024-03-19T01:23:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd440a9020 con 0x7fdd340039c0 2024-03-19T01:23:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd3ffff640 1 -- 172.21.15.7:0/4141575097 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd440058e0 con 0x7fdd340039c0 2024-03-19T01:23:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4f48d640 1 -- 172.21.15.7:0/4141575097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd340039c0 msgr2=0x7fdd34003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4f48d640 1 --2- 172.21.15.7:0/4141575097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd340039c0 0x7fdd34003e00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c290 tx=0x7fdd4401cd20 comp rx=0 tx=0).stop 2024-03-19T01:23:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4da2a640 1 -- 172.21.15.7:0/4141575097 reap_dead start 2024-03-19T01:23:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4f48d640 1 -- 172.21.15.7:0/4141575097 shutdown_connections 2024-03-19T01:23:51.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4f48d640 1 -- 172.21.15.7:0/4141575097 >> 172.21.15.7:0/4141575097 conn(0x7fdd480b6510 msgr2=0x7fdd480b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:51.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4f48d640 1 -- 172.21.15.7:0/4141575097 shutdown_connections 2024-03-19T01:23:51.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4f48d640 1 -- 172.21.15.7:0/4141575097 wait complete. 2024-03-19T01:23:51.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4f48d640 1 Processor -- start 2024-03-19T01:23:51.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.723+0000 7fdd4f48d640 1 -- start start 2024-03-19T01:23:51.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4f48d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd340039c0 0x7fdd48144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4f48d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd4809ac50 0x7fdd481db850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4f48d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd4813eb10 0x7fdd4813ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4f48d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48148580 con 0x7fdd4809ac50 2024-03-19T01:23:51.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4f48d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48070ac0 con 0x7fdd4813eb10 2024-03-19T01:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4f48d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdd4814e400 con 0x7fdd340039c0 2024-03-19T01:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4ca28640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd4809ac50 0x7fdd481db850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4ca28640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd4809ac50 0x7fdd481db850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43092/0 (socket says 172.21.15.7:43092) 2024-03-19T01:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4ca28640 1 -- 172.21.15.7:0/1017747993 learned_addr learned my addr 172.21.15.7:0/1017747993 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4d229640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd340039c0 0x7fdd48144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4da2a640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd4813eb10 0x7fdd4813ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4ca28640 1 -- 172.21.15.7:0/1017747993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd340039c0 msgr2=0x7fdd48144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4ca28640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd340039c0 0x7fdd48144ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4ca28640 1 -- 172.21.15.7:0/1017747993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd4813eb10 msgr2=0x7fdd4813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4ca28640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd4813eb10 0x7fdd4813ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4ca28640 1 -- 172.21.15.7:0/1017747993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4809c1c0 con 0x7fdd4809ac50 2024-03-19T01:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4ca28640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd4809ac50 0x7fdd481db850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdd38011590 tx=0x7fdd38011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd3dffb640 1 -- 172.21.15.7:0/1017747993 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38004220 con 0x7fdd4809ac50 2024-03-19T01:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd3dffb640 1 -- 172.21.15.7:0/1017747993 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd380043c0 con 0x7fdd4809ac50 2024-03-19T01:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd480b2b80 con 0x7fdd4809ac50 2024-03-19T01:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd3dffb640 1 -- 172.21.15.7:0/1017747993 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38015870 con 0x7fdd4809ac50 2024-03-19T01:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd4813f4d0 con 0x7fdd4809ac50 2024-03-19T01:23:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd48075460 con 0x7fdd4809ac50 2024-03-19T01:23:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd3dffb640 1 -- 172.21.15.7:0/1017747993 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdd38015a10 con 0x7fdd4809ac50 2024-03-19T01:23:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd3dffb640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdd2c067690 0x7fdd2c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd3dffb640 1 -- 172.21.15.7:0/1017747993 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdd38085c50 con 0x7fdd4809ac50 2024-03-19T01:23:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4d229640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdd2c067690 0x7fdd2c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.727+0000 7fdd4d229640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdd2c067690 0x7fdd2c069b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdd440aa000 tx=0x7fdd44001a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.731+0000 7fdd3dffb640 1 -- 172.21.15.7:0/1017747993 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdd3808b020 con 0x7fdd4809ac50 2024-03-19T01:23:51.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:51.955+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]} v 0) -- 0x7fdd48144770 con 0x7fdd4809ac50 2024-03-19T01:23:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.651+0000 7fdd3dffb640 1 -- 172.21.15.7:0/1017747993 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]}]=0 v49) ==== 137+0+0 (secure 0 0 0) 0x7fdd38019030 con 0x7fdd4809ac50 2024-03-19T01:23:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdd2c067690 msgr2=0x7fdd2c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdd2c067690 0x7fdd2c069b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdd440aa000 tx=0x7fdd44001a50 comp rx=0 tx=0).stop 2024-03-19T01:23:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd4809ac50 msgr2=0x7fdd481db850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd4809ac50 0x7fdd481db850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdd38011590 tx=0x7fdd38011a50 comp rx=0 tx=0).stop 2024-03-19T01:23:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 shutdown_connections 2024-03-19T01:23:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdd2c067690 0x7fdd2c069b50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd4813eb10 0x7fdd4813ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd4809ac50 0x7fdd481db850 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 --2- 172.21.15.7:0/1017747993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd340039c0 0x7fdd48144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 >> 172.21.15.7:0/1017747993 conn(0x7fdd480b6510 msgr2=0x7fdd48067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 shutdown_connections 2024-03-19T01:23:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.655+0000 7fdd4f48d640 1 -- 172.21.15.7:0/1017747993 wait complete. 2024-03-19T01:23:52.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-19T01:23:52.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c99e4640 1 Processor -- start 2024-03-19T01:23:52.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c99e4640 1 -- start start 2024-03-19T01:23:52.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c99e4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70c40bd400 0x7f70c40b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:52.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c99e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 0x7f70c414ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:52.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c99e4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70c40bd7e0 con 0x7f70c414a960 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c99e4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f70c40b4f20 con 0x7f70c40bd400 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c99e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f70c40b6d70 con 0x7f70c4141650 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c2ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70c40bd400 0x7f70c40b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c2ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70c40bd400 0x7f70c40b57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49366/0 (socket says 172.21.15.7:49366) 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c2ffd640 1 -- 172.21.15.7:0/1857023706 learned_addr learned my addr 172.21.15.7:0/1857023706 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c37fe640 1 --2- 172.21.15.7:0/1857023706 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 0x7f70c414ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c2ffd640 1 -- 172.21.15.7:0/1857023706 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 msgr2=0x7f70c414ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c2ffd640 1 --2- 172.21.15.7:0/1857023706 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 0x7f70c414ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c2ffd640 1 -- 172.21.15.7:0/1857023706 >> v1:172.21.15.167:6789/0 conn(0x7f70c414a960 legacy=0x7f70c4140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c2ffd640 1 -- 172.21.15.7:0/1857023706 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c40b5ce0 con 0x7f70c40bd400 2024-03-19T01:23:52.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c2ffd640 1 --2- 172.21.15.7:0/1857023706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c40bd400 0x7f70c40b57a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f70b40098a0 tx=0x7f70b401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47c646fd35d1eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.799+0000 7f70c27fc640 1 -- 172.21.15.7:0/1857023706 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b40b0070 con 0x7f70c40bd400 2024-03-19T01:23:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c27fc640 1 -- 172.21.15.7:0/1857023706 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b40ab600 con 0x7f70c40bd400 2024-03-19T01:23:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c27fc640 1 -- 172.21.15.7:0/1857023706 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b40ab9d0 con 0x7f70c40bd400 2024-03-19T01:23:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- 172.21.15.7:0/1857023706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c40bd400 msgr2=0x7f70c40b57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 --2- 172.21.15.7:0/1857023706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c40bd400 0x7f70c40b57a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f70b40098a0 tx=0x7f70b401c470 comp rx=0 tx=0).stop 2024-03-19T01:23:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- 172.21.15.7:0/1857023706 shutdown_connections 2024-03-19T01:23:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 --2- 172.21.15.7:0/1857023706 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 0x7f70c414ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 --2- 172.21.15.7:0/1857023706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c40bd400 0x7f70c40b57a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- 172.21.15.7:0/1857023706 >> 172.21.15.7:0/1857023706 conn(0x7f70c4072df0 msgr2=0x7f70c4073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- 172.21.15.7:0/1857023706 shutdown_connections 2024-03-19T01:23:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- 172.21.15.7:0/1857023706 wait complete. 2024-03-19T01:23:52.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 Processor -- start 2024-03-19T01:23:52.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- start start 2024-03-19T01:23:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40bd400 0x7f70c40bc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 0x7f70c4142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c414a960 0x7f70c41e2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f70c4154510 con 0x7f70c414a960 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f70c40b4a90 con 0x7f70c40bd400 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f70c4154690 con 0x7f70c4141650 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c414a960 0x7f70c41e2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c414a960 0x7f70c41e2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49370/0 (socket says 172.21.15.7:49370) 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c37fe640 1 -- 172.21.15.7:0/3852776203 learned_addr learned my addr 172.21.15.7:0/3852776203 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70bbfff640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 0x7f70c4142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c2ffd640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40bd400 0x7f70c40bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c37fe640 1 -- 172.21.15.7:0/3852776203 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 msgr2=0x7f70c4142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c37fe640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 0x7f70c4142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c37fe640 1 -- 172.21.15.7:0/3852776203 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40bd400 msgr2=0x7f70c40bc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c37fe640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40bd400 0x7f70c40bc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c37fe640 1 -- 172.21.15.7:0/3852776203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c41488f0 con 0x7f70c414a960 2024-03-19T01:23:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c37fe640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c414a960 0x7f70c41e2f70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00cd90 tx=0x7f70bc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c0ff9640 1 -- 172.21.15.7:0/3852776203 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc003ba0 con 0x7f70c414a960 2024-03-19T01:23:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c0ff9640 1 -- 172.21.15.7:0/3852776203 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70bc003d60 con 0x7f70c414a960 2024-03-19T01:23:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c4148bd0 con 0x7f70c414a960 2024-03-19T01:23:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c0ff9640 1 -- 172.21.15.7:0/3852776203 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc004260 con 0x7f70c414a960 2024-03-19T01:23:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.803+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c4144250 con 0x7f70c414a960 2024-03-19T01:23:52.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.807+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c4075570 con 0x7f70c414a960 2024-03-19T01:23:52.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.807+0000 7f70c0ff9640 1 -- 172.21.15.7:0/3852776203 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f70bc004400 con 0x7f70c414a960 2024-03-19T01:23:52.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.807+0000 7f70c0ff9640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f70a4067660 0x7f70a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:52.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.807+0000 7f70c0ff9640 1 -- 172.21.15.7:0/3852776203 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f70bc086290 con 0x7f70c414a960 2024-03-19T01:23:52.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.807+0000 7f70c2ffd640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f70a4067660 0x7f70a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:52.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.807+0000 7f70c2ffd640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f70a4067660 0x7f70a4069b20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f70b40ad000 tx=0x7f70b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:52.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.811+0000 7f70c0ff9640 1 -- 172.21.15.7:0/3852776203 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f70bc01d340 con 0x7f70c414a960 2024-03-19T01:23:52.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.967+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f70c4145a40 con 0x7f70c414a960 2024-03-19T01:23:52.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c0ff9640 1 -- 172.21.15.7:0/3852776203 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f70bc056fa0 con 0x7f70c414a960 2024-03-19T01:23:52.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f70a4067660 msgr2=0x7f70a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:52.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f70a4067660 0x7f70a4069b20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f70b40ad000 tx=0x7f70b4008040 comp rx=0 tx=0).stop 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c414a960 msgr2=0x7f70c41e2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c414a960 0x7f70c41e2f70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00cd90 tx=0x7f70bc00a660 comp rx=0 tx=0).stop 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 shutdown_connections 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f70a4067660 0x7f70a4069b20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c4141650 0x7f70c4142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70c414a960 0x7f70c41e2f70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 --2- 172.21.15.7:0/3852776203 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40bd400 0x7f70c40bc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 >> 172.21.15.7:0/3852776203 conn(0x7f70c4072df0 msgr2=0x7f70c4140d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 shutdown_connections 2024-03-19T01:23:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:52.971+0000 7f70c99e4640 1 -- 172.21.15.7:0/3852776203 wait complete. 2024-03-19T01:23:52.989 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-19T01:23:52.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-19T01:23:53.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:23:53.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:23:53.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-19T01:23:53.086 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-19T01:23:53.086 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:23:53.086 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:23:53.120 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:23:53.120 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:23:53.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:53.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.323+0000 7f7c89cb2640 1 Processor -- start 2024-03-19T01:23:53.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.323+0000 7f7c89cb2640 1 -- start start 2024-03-19T01:23:53.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.323+0000 7f7c89cb2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c840b5f10 0x7f7c8413deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.323+0000 7f7c89cb2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c84144170 con 0x7f7c8413e3f0 2024-03-19T01:23:53.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.323+0000 7f7c89cb2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c840ba090 con 0x7f7c840b9cb0 2024-03-19T01:23:53.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.323+0000 7f7c89cb2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f7c84144a40 con 0x7f7c840b5f10 2024-03-19T01:23:53.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.323+0000 7f7c82ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c840b5f10 0x7f7c8413deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:53.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.323+0000 7f7c83fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7c8413e3f0 0x7f7c8414ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35132/0 (socket says 172.21.15.7:35132) 2024-03-19T01:23:53.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.323+0000 7f7c83fff640 1 -- 172.21.15.7:0/3584255495 learned_addr learned my addr 172.21.15.7:0/3584255495 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:53.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c82ffd640 1 -- 172.21.15.7:0/3584255495 >> v1:172.21.15.167:6789/0 conn(0x7f7c840b9cb0 legacy=0x7f7c84147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:53.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c82ffd640 1 -- 172.21.15.7:0/3584255495 >> v1:172.21.15.7:6789/0 conn(0x7f7c8413e3f0 legacy=0x7f7c8414ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c827fc640 1 -- 172.21.15.7:0/3584255495 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 922438059 0 0) 0x7f7c7c0014a0 con 0x7f7c8413e3f0 2024-03-19T01:23:53.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c82ffd640 1 -- 172.21.15.7:0/3584255495 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c840726e0 con 0x7f7c840b5f10 2024-03-19T01:23:53.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c82ffd640 1 --2- 172.21.15.7:0/3584255495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c840b5f10 0x7f7c8413deb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0097c0 tx=0x7f7c6c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4d938196eb9b05 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:53.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c827fc640 1 -- 172.21.15.7:0/3584255495 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0b0070 con 0x7f7c840b5f10 2024-03-19T01:23:53.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c827fc640 1 -- 172.21.15.7:0/3584255495 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c6c0ab600 con 0x7f7c840b5f10 2024-03-19T01:23:53.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c827fc640 1 -- 172.21.15.7:0/3584255495 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0ab9d0 con 0x7f7c840b5f10 2024-03-19T01:23:53.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3584255495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c840b5f10 msgr2=0x7f7c8413deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c89cb2640 1 --2- 172.21.15.7:0/3584255495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c840b5f10 0x7f7c8413deb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0097c0 tx=0x7f7c6c01c3d0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3584255495 shutdown_connections 2024-03-19T01:23:53.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c89cb2640 1 --2- 172.21.15.7:0/3584255495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c840b5f10 0x7f7c8413deb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3584255495 >> 172.21.15.7:0/3584255495 conn(0x7f7c84074130 msgr2=0x7f7c84074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:53.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3584255495 shutdown_connections 2024-03-19T01:23:53.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.327+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3584255495 wait complete. 2024-03-19T01:23:53.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 Processor -- start 2024-03-19T01:23:53.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 -- start start 2024-03-19T01:23:53.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c840b5f10 0x7f7c84148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c840b9cb0 0x7f7c840b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c8413e3f0 0x7f7c841499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7c84145050 con 0x7f7c8413e3f0 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7c84144ed0 con 0x7f7c840b5f10 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7c841529d0 con 0x7f7c840b9cb0 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c8413e3f0 0x7f7c841499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c8413e3f0 0x7f7c841499c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49390/0 (socket says 172.21.15.7:49390) 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c83fff640 1 -- 172.21.15.7:0/3950416544 learned_addr learned my addr 172.21.15.7:0/3950416544 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c837fe640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c840b5f10 0x7f7c84148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c82ffd640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c840b9cb0 0x7f7c840b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c83fff640 1 -- 172.21.15.7:0/3950416544 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c840b9cb0 msgr2=0x7f7c840b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c83fff640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c840b9cb0 0x7f7c840b0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c83fff640 1 -- 172.21.15.7:0/3950416544 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c840b5f10 msgr2=0x7f7c84148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c83fff640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c840b5f10 0x7f7c84148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c83fff640 1 -- 172.21.15.7:0/3950416544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c840b23f0 con 0x7f7c8413e3f0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c837fe640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c840b5f10 0x7f7c84148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c83fff640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c8413e3f0 0x7f7c841499c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00c730 tx=0x7f7c7c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c7bfff640 1 -- 172.21.15.7:0/3950416544 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c018070 con 0x7f7c8413e3f0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c7bfff640 1 -- 172.21.15.7:0/3950416544 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c7c00ce40 con 0x7f7c8413e3f0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c840b0eb0 con 0x7f7c8413e3f0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c7bfff640 1 -- 172.21.15.7:0/3950416544 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c00a860 con 0x7f7c8413e3f0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.331+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c84148820 con 0x7f7c8413e3f0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.335+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c84075460 con 0x7f7c8413e3f0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.335+0000 7f7c7bfff640 1 -- 172.21.15.7:0/3950416544 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7c7c01b070 con 0x7f7c8413e3f0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.335+0000 7f7c7bfff640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7c64067660 0x7f7c64069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.335+0000 7f7c837fe640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7c64067660 0x7f7c64069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:53.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.335+0000 7f7c7bfff640 1 -- 172.21.15.7:0/3950416544 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7c7c084d60 con 0x7f7c8413e3f0 2024-03-19T01:23:53.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.335+0000 7f7c837fe640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7c64067660 0x7f7c64069b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7c74003e40 tx=0x7f7c74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:53.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.339+0000 7f7c7bfff640 1 -- 172.21.15.7:0/3950416544 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7c84144170 con 0x7f7c8413e3f0 2024-03-19T01:23:53.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.555+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c840b7d20 con 0x7f7c8413e3f0 2024-03-19T01:23:53.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.555+0000 7f7c7bfff640 1 -- 172.21.15.7:0/3950416544 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7f7c7c055a70 con 0x7f7c8413e3f0 2024-03-19T01:23:53.558 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:53.558 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7c64067660 msgr2=0x7f7c64069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7c64067660 0x7f7c64069b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7c74003e40 tx=0x7f7c74009040 comp rx=0 tx=0).stop 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c8413e3f0 msgr2=0x7f7c841499c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c8413e3f0 0x7f7c841499c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00c730 tx=0x7f7c7c00cbf0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 shutdown_connections 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7c64067660 0x7f7c64069b20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c840b9cb0 0x7f7c840b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c8413e3f0 0x7f7c841499c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 --2- 172.21.15.7:0/3950416544 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c840b5f10 0x7f7c84148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 >> 172.21.15.7:0/3950416544 conn(0x7f7c84074130 msgr2=0x7f7c841424a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 shutdown_connections 2024-03-19T01:23:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.559+0000 7f7c89cb2640 1 -- 172.21.15.7:0/3950416544 wait complete. 2024-03-19T01:23:53.577 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:53.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:53.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.691+0000 7fe2a4d78640 1 Processor -- start 2024-03-19T01:23:53.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.691+0000 7fe2a4d78640 1 -- start start 2024-03-19T01:23:53.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.691+0000 7fe2a4d78640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2a0074390 0x7fe2a0145400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.691+0000 7fe2a4d78640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2a0142e70 con 0x7fe2a0142a60 2024-03-19T01:23:53.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.691+0000 7fe2a4d78640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2a0074770 con 0x7fe2a00b3070 2024-03-19T01:23:53.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.691+0000 7fe2a4d78640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fe2a0071110 con 0x7fe2a0074390 2024-03-19T01:23:53.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.691+0000 7fe29ed9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe2a0142a60 0x7fe2a014fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35144/0 (socket says 172.21.15.7:35144) 2024-03-19T01:23:53.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.691+0000 7fe29ed9d640 1 -- 172.21.15.7:0/2530459208 learned_addr learned my addr 172.21.15.7:0/2530459208 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:53.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29e59c640 1 --2- 172.21.15.7:0/2530459208 >> v2:172.21.15.7:3300/0 conn(0x7fe2a0074390 0x7fe2a0145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:53.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29dd9b640 1 -- 172.21.15.7:0/2530459208 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590650529 0 0) 0x7fe2a0142e70 con 0x7fe2a0142a60 2024-03-19T01:23:53.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29dd9b640 1 -- 172.21.15.7:0/2530459208 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe28c003530 con 0x7fe2a0142a60 2024-03-19T01:23:53.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29e59c640 1 -- 172.21.15.7:0/2530459208 >> v1:172.21.15.167:6789/0 conn(0x7fe2a00b3070 legacy=0x7fe2a0142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29dd9b640 1 -- 172.21.15.7:0/2530459208 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2574670306 0 0) 0x7fe28c003530 con 0x7fe2a0142a60 2024-03-19T01:23:53.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29e59c640 1 -- 172.21.15.7:0/2530459208 >> v1:172.21.15.7:6789/0 conn(0x7fe2a0142a60 legacy=0x7fe2a014fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29e59c640 1 -- 172.21.15.7:0/2530459208 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a00b5680 con 0x7fe2a0074390 2024-03-19T01:23:53.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29e59c640 1 --2- 172.21.15.7:0/2530459208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 0x7fe2a0145400 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe2900097c0 tx=0x7fe29001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9527f7f59172b201 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:53.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29dd9b640 1 -- 172.21.15.7:0/2530459208 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2900b0070 con 0x7fe2a0074390 2024-03-19T01:23:53.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29dd9b640 1 -- 172.21.15.7:0/2530459208 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2900ab400 con 0x7fe2a0074390 2024-03-19T01:23:53.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe29dd9b640 1 -- 172.21.15.7:0/2530459208 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298001a00 con 0x7fe2a0074390 2024-03-19T01:23:53.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe2a4d78640 1 -- 172.21.15.7:0/2530459208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 msgr2=0x7fe2a0145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe2a4d78640 1 --2- 172.21.15.7:0/2530459208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 0x7fe2a0145400 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe2900097c0 tx=0x7fe29001c3c0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe2a4d78640 1 -- 172.21.15.7:0/2530459208 shutdown_connections 2024-03-19T01:23:53.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe2a4d78640 1 --2- 172.21.15.7:0/2530459208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 0x7fe2a0145400 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe2a4d78640 1 -- 172.21.15.7:0/2530459208 >> 172.21.15.7:0/2530459208 conn(0x7fe2a00b9120 msgr2=0x7fe2a00b9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:53.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe2a4d78640 1 -- 172.21.15.7:0/2530459208 shutdown_connections 2024-03-19T01:23:53.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.695+0000 7fe2a4d78640 1 -- 172.21.15.7:0/2530459208 wait complete. 2024-03-19T01:23:53.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 Processor -- start 2024-03-19T01:23:53.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 -- start start 2024-03-19T01:23:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 0x7fe2a00b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2a00b3070 0x7fe2a01e0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2a0142a60 0x7fe2a01e0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a0070e20 con 0x7fe2a0074390 2024-03-19T01:23:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a0070ca0 con 0x7fe2a0142a60 2024-03-19T01:23:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a0153be0 con 0x7fe2a00b3070 2024-03-19T01:23:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe29e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 0x7fe2a00b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe29e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 0x7fe2a00b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49402/0 (socket says 172.21.15.7:49402) 2024-03-19T01:23:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe29e59c640 1 -- 172.21.15.7:0/3925919596 learned_addr learned my addr 172.21.15.7:0/3925919596 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe29e59c640 1 -- 172.21.15.7:0/3925919596 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2a00b3070 msgr2=0x7fe2a01e0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe29e59c640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2a00b3070 0x7fe2a01e0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe29e59c640 1 -- 172.21.15.7:0/3925919596 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2a0142a60 msgr2=0x7fe2a01e0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe29e59c640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2a0142a60 0x7fe2a01e0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe29e59c640 1 -- 172.21.15.7:0/3925919596 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a0148300 con 0x7fe2a0074390 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe29e59c640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 0x7fe2a00b1a00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe2900ae040 tx=0x7fe29001ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe296ffd640 1 -- 172.21.15.7:0/3925919596 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2900bb020 con 0x7fe2a0074390 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe296ffd640 1 -- 172.21.15.7:0/3925919596 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2900b1040 con 0x7fe2a0074390 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a0146be0 con 0x7fe2a0074390 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe296ffd640 1 -- 172.21.15.7:0/3925919596 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2900b0050 con 0x7fe2a0074390 2024-03-19T01:23:53.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a0148020 con 0x7fe2a0074390 2024-03-19T01:23:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a0075570 con 0x7fe2a0074390 2024-03-19T01:23:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.699+0000 7fe296ffd640 1 -- 172.21.15.7:0/3925919596 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe2900be070 con 0x7fe2a0074390 2024-03-19T01:23:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.703+0000 7fe296ffd640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe280067660 0x7fe280069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.703+0000 7fe296ffd640 1 -- 172.21.15.7:0/3925919596 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe29011f0f0 con 0x7fe2a0074390 2024-03-19T01:23:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.703+0000 7fe297fff640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe280067660 0x7fe280069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.703+0000 7fe297fff640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe280067660 0x7fe280069b20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe288000f70 tx=0x7fe2880079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:53.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.707+0000 7fe296ffd640 1 -- 172.21.15.7:0/3925919596 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe2900c4080 con 0x7fe2a0074390 2024-03-19T01:23:53.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.863+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe2a00b8ae0 con 0x7fe2a0074390 2024-03-19T01:23:53.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.863+0000 7fe296ffd640 1 -- 172.21.15.7:0/3925919596 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fe290015fb0 con 0x7fe2a0074390 2024-03-19T01:23:53.866 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:53.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe280067660 msgr2=0x7fe280069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe280067660 0x7fe280069b20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe288000f70 tx=0x7fe2880079c0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 msgr2=0x7fe2a00b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 0x7fe2a00b1a00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe2900ae040 tx=0x7fe29001ac00 comp rx=0 tx=0).stop 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 shutdown_connections 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe280067660 0x7fe280069b20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2a00b3070 0x7fe2a01e0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2a0142a60 0x7fe2a01e0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 --2- 172.21.15.7:0/3925919596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2a0074390 0x7fe2a00b1a00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 >> 172.21.15.7:0/3925919596 conn(0x7fe2a00b9120 msgr2=0x7fe2a00b8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 shutdown_connections 2024-03-19T01:23:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.867+0000 7fe2a4d78640 1 -- 172.21.15.7:0/3925919596 wait complete. 2024-03-19T01:23:53.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.999+0000 7f8624cad640 1 Processor -- start 2024-03-19T01:23:54.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:53.999+0000 7f8624cad640 1 -- start start 2024-03-19T01:23:54.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f8624cad640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f862006df70 0x7f862006eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f8624cad640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f862006c0e0 con 0x7f86200b5890 2024-03-19T01:23:54.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f8624cad640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86200b5c70 con 0x7f862009ed60 2024-03-19T01:23:54.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f8624cad640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f862006e3c0 con 0x7f862006df70 2024-03-19T01:23:54.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861e59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f86200b5890 0x7f862006b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35152/0 (socket says 172.21.15.7:35152) 2024-03-19T01:23:54.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861e59c640 1 -- 172.21.15.7:0/2947718812 learned_addr learned my addr 172.21.15.7:0/2947718812 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:54.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861ed9d640 1 --2- 172.21.15.7:0/2947718812 >> v2:172.21.15.167:3300/0 conn(0x7f862006df70 0x7f862006eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1628988000 0 0) 0x7f862006c0e0 con 0x7f86200b5890 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8608003530 con 0x7f86200b5890 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4221528010 0 0) 0x7f8608003530 con 0x7f86200b5890 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 >> v2:172.21.15.167:3300/0 conn(0x7f862006df70 msgr2=0x7f862006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 --2- 172.21.15.7:0/2947718812 >> v2:172.21.15.167:3300/0 conn(0x7f862006df70 0x7f862006eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 >> v1:172.21.15.167:6789/0 conn(0x7f862009ed60 legacy=0x7f862006b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86200711c0 con 0x7f86200b5890 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f8614001680 con 0x7f86200b5890 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 >> v1:172.21.15.7:6789/0 conn(0x7f86200b5890 legacy=0x7f862006b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86080039c0 0x7f8608003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8608004810 0x7f8608004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86080052f0 0x7f860800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f862006e3c0 con 0x7f86080039c0 2024-03-19T01:23:54.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f860800c1c0 con 0x7f86080052f0 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f860800c3c0 con 0x7f8608004810 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861e59c640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86080039c0 0x7f8608003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861dd9b640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8608004810 0x7f8608004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861ed9d640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86080052f0 0x7f860800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861e59c640 1 -- 172.21.15.7:0/2947718812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8608004810 msgr2=0x7f8608004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861e59c640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8608004810 0x7f8608004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861e59c640 1 -- 172.21.15.7:0/2947718812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86080052f0 msgr2=0x7f860800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861e59c640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86080052f0 0x7f860800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861e59c640 1 -- 172.21.15.7:0/2947718812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8608003530 con 0x7f86080039c0 2024-03-19T01:23:54.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.003+0000 7f861ed9d640 1 -- 172.21.15.7:0/2947718812 reap_dead start 2024-03-19T01:23:54.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861e59c640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86080039c0 0x7f8608003de0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f861400c190 tx=0x7f861401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479ee4f60319612c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614007d00 con 0x7f86080039c0 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86140a9020 con 0x7f86080039c0 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861d59a640 1 -- 172.21.15.7:0/2947718812 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614005ae0 con 0x7f86080039c0 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- 172.21.15.7:0/2947718812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86080039c0 msgr2=0x7f8608003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86080039c0 0x7f8608003de0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f861400c190 tx=0x7f861401cc00 comp rx=0 tx=0).stop 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- 172.21.15.7:0/2947718812 shutdown_connections 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 --2- 172.21.15.7:0/2947718812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86080039c0 0x7f8608003de0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- 172.21.15.7:0/2947718812 >> 172.21.15.7:0/2947718812 conn(0x7f86200b68f0 msgr2=0x7f86200b6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- 172.21.15.7:0/2947718812 shutdown_connections 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- 172.21.15.7:0/2947718812 wait complete. 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 Processor -- start 2024-03-19T01:23:54.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- start start 2024-03-19T01:23:54.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86080039c0 0x7f8620148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86200b0ab0 0x7f86201e3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86200aef80 0x7f86201d3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8620070ee0 con 0x7f86200aef80 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f862009f140 con 0x7f86200b0ab0 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f862006bad0 con 0x7f86080039c0 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86200aef80 0x7f86201d3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86200aef80 0x7f86201d3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49422/0 (socket says 172.21.15.7:49422) 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861ed9d640 1 -- 172.21.15.7:0/3604293790 learned_addr learned my addr 172.21.15.7:0/3604293790 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861e59c640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86080039c0 0x7f8620148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861dd9b640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86200b0ab0 0x7f86201e3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861ed9d640 1 -- 172.21.15.7:0/3604293790 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86080039c0 msgr2=0x7f8620148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861ed9d640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86080039c0 0x7f8620148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861ed9d640 1 -- 172.21.15.7:0/3604293790 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86200b0ab0 msgr2=0x7f86201e3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861ed9d640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86200b0ab0 0x7f86201e3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861ed9d640 1 -- 172.21.15.7:0/3604293790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86201e3670 con 0x7f86200aef80 2024-03-19T01:23:54.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861dd9b640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86200b0ab0 0x7f86201e3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:54.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f861ed9d640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86200aef80 0x7f86201d3370 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8620070ab0 tx=0x7f86180115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:54.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f860effd640 1 -- 172.21.15.7:0/3604293790 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618011e10 con 0x7f86200aef80 2024-03-19T01:23:54.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f860effd640 1 -- 172.21.15.7:0/3604293790 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f861801b070 con 0x7f86200aef80 2024-03-19T01:23:54.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86200b02f0 con 0x7f86200aef80 2024-03-19T01:23:54.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f860effd640 1 -- 172.21.15.7:0/3604293790 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86180039a0 con 0x7f86200aef80 2024-03-19T01:23:54.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.007+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86201d1a60 con 0x7f86200aef80 2024-03-19T01:23:54.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.011+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f862006c3d0 con 0x7f86200aef80 2024-03-19T01:23:54.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.011+0000 7f860effd640 1 -- 172.21.15.7:0/3604293790 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8618003b40 con 0x7f86200aef80 2024-03-19T01:23:54.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.011+0000 7f860effd640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8600067660 0x7f8600069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.011+0000 7f860effd640 1 -- 172.21.15.7:0/3604293790 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8618085f30 con 0x7f86200aef80 2024-03-19T01:23:54.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.011+0000 7f861e59c640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8600067660 0x7f8600069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.011+0000 7f861e59c640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8600067660 0x7f8600069b20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f86140aa000 tx=0x7f862006d890 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:54.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.015+0000 7f860effd640 1 -- 172.21.15.7:0/3604293790 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8618056c40 con 0x7f86200aef80 2024-03-19T01:23:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.167+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8620070c00 con 0x7f86200aef80 2024-03-19T01:23:54.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.167+0000 7f860effd640 1 -- 172.21.15.7:0/3604293790 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f8618056540 con 0x7f86200aef80 2024-03-19T01:23:54.170 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:54.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8600067660 msgr2=0x7f8600069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8600067660 0x7f8600069b20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f86140aa000 tx=0x7f862006d890 comp rx=0 tx=0).stop 2024-03-19T01:23:54.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86200aef80 msgr2=0x7f86201d3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86200aef80 0x7f86201d3370 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8620070ab0 tx=0x7f86180115a0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 shutdown_connections 2024-03-19T01:23:54.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8600067660 0x7f8600069b20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86200aef80 0x7f86201d3370 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86200b0ab0 0x7f86201e3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 --2- 172.21.15.7:0/3604293790 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86080039c0 0x7f8620148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 >> 172.21.15.7:0/3604293790 conn(0x7f86200b68f0 msgr2=0x7f8620146330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 shutdown_connections 2024-03-19T01:23:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.171+0000 7f8624cad640 1 -- 172.21.15.7:0/3604293790 wait complete. 2024-03-19T01:23:54.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:54.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.299+0000 7f040bcb4640 1 Processor -- start 2024-03-19T01:23:54.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.299+0000 7f040bcb4640 1 -- start start 2024-03-19T01:23:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040bcb4640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f04040b9bb0 0x7f04040afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040bcb4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04040af4f0 0x7f0404147670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040bcb4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 0x7f040413dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040bcb4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f04040b5e30 con 0x7f04040af4f0 2024-03-19T01:23:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040bcb4640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f04040b9f90 con 0x7f04040b9bb0 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040bcb4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f04040b62c0 con 0x7f0404147bb0 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040924f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04040af4f0 0x7f0404147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040924f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04040af4f0 0x7f0404147670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49424/0 (socket says 172.21.15.7:49424) 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040924f640 1 -- 172.21.15.7:0/2739281021 learned_addr learned my addr 172.21.15.7:0/2739281021 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040a251640 1 --2- 172.21.15.7:0/2739281021 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 0x7f040413dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f0409a50640 1 --2- 172.21.15.7:0/2739281021 >> v2:172.21.15.167:3300/0 conn(0x7f04040b9bb0 0x7f04040afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040924f640 1 -- 172.21.15.7:0/2739281021 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 msgr2=0x7f040413dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040924f640 1 --2- 172.21.15.7:0/2739281021 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 0x7f040413dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040924f640 1 -- 172.21.15.7:0/2739281021 >> v2:172.21.15.167:3300/0 conn(0x7f04040b9bb0 msgr2=0x7f04040afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040924f640 1 --2- 172.21.15.7:0/2739281021 >> v2:172.21.15.167:3300/0 conn(0x7f04040b9bb0 0x7f04040afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.303+0000 7f040924f640 1 -- 172.21.15.7:0/2739281021 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04040b5ac0 con 0x7f04040af4f0 2024-03-19T01:23:54.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040924f640 1 --2- 172.21.15.7:0/2739281021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040af4f0 0x7f0404147670 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f04040ba750 tx=0x7f03f401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13b27e7008780c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:54.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f0408a4e640 1 -- 172.21.15.7:0/2739281021 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f40b0070 con 0x7f04040af4f0 2024-03-19T01:23:54.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f0408a4e640 1 -- 172.21.15.7:0/2739281021 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f40a4e50 con 0x7f04040af4f0 2024-03-19T01:23:54.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f0408a4e640 1 -- 172.21.15.7:0/2739281021 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f40ab820 con 0x7f04040af4f0 2024-03-19T01:23:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040bcb4640 1 -- 172.21.15.7:0/2739281021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040af4f0 msgr2=0x7f0404147670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2739281021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040af4f0 0x7f0404147670 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f04040ba750 tx=0x7f03f401c400 comp rx=0 tx=0).stop 2024-03-19T01:23:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040bcb4640 1 -- 172.21.15.7:0/2739281021 shutdown_connections 2024-03-19T01:23:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2739281021 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 0x7f040413dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2739281021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040af4f0 0x7f0404147670 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2739281021 >> v2:172.21.15.167:3300/0 conn(0x7f04040b9bb0 0x7f04040afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040bcb4640 1 -- 172.21.15.7:0/2739281021 >> 172.21.15.7:0/2739281021 conn(0x7f040406e1b0 msgr2=0x7f040406e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040bcb4640 1 -- 172.21.15.7:0/2739281021 shutdown_connections 2024-03-19T01:23:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.307+0000 7f040bcb4640 1 -- 172.21.15.7:0/2739281021 wait complete. 2024-03-19T01:23:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 Processor -- start 2024-03-19T01:23:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 -- start start 2024-03-19T01:23:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04040af4f0 0x7f040406b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040b9bb0 0x7f0404138020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 0x7f0404143a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f04040b4f80 con 0x7f04040b9bb0 2024-03-19T01:23:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f04040b4e00 con 0x7f04040af4f0 2024-03-19T01:23:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f04040b5100 con 0x7f0404147bb0 2024-03-19T01:23:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040924f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040b9bb0 0x7f0404138020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040924f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040b9bb0 0x7f0404138020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49440/0 (socket says 172.21.15.7:49440) 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040924f640 1 -- 172.21.15.7:0/2504411161 learned_addr learned my addr 172.21.15.7:0/2504411161 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040a251640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 0x7f0404143a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f0409a50640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04040af4f0 0x7f040406b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040924f640 1 -- 172.21.15.7:0/2504411161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 msgr2=0x7f0404143a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040924f640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 0x7f0404143a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040924f640 1 -- 172.21.15.7:0/2504411161 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04040af4f0 msgr2=0x7f040406b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040924f640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04040af4f0 0x7f040406b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040924f640 1 -- 172.21.15.7:0/2504411161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040413fea0 con 0x7f04040b9bb0 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040924f640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040b9bb0 0x7f0404138020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f03f401aab0 tx=0x7f03f40191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f03fa7fc640 1 -- 172.21.15.7:0/2504411161 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f40b0040 con 0x7f04040b9bb0 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f03fa7fc640 1 -- 172.21.15.7:0/2504411161 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f40aa660 con 0x7f04040b9bb0 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f03fa7fc640 1 -- 172.21.15.7:0/2504411161 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f40aaa50 con 0x7f04040b9bb0 2024-03-19T01:23:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0404144e20 con 0x7f04040b9bb0 2024-03-19T01:23:54.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f040413fbc0 con 0x7f04040b9bb0 2024-03-19T01:23:54.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0404075570 con 0x7f04040b9bb0 2024-03-19T01:23:54.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.311+0000 7f03fa7fc640 1 -- 172.21.15.7:0/2504411161 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f03f40aabf0 con 0x7f04040b9bb0 2024-03-19T01:23:54.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.315+0000 7f03fa7fc640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03e8067660 0x7f03e8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.315+0000 7f03fa7fc640 1 -- 172.21.15.7:0/2504411161 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f03f411f6c0 con 0x7f04040b9bb0 2024-03-19T01:23:54.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.315+0000 7f0409a50640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03e8067660 0x7f03e8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.315+0000 7f0409a50640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03e8067660 0x7f03e8069b20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0400009790 tx=0x7f0400008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:54.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.319+0000 7f03fa7fc640 1 -- 172.21.15.7:0/2504411161 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f03f4016110 con 0x7f04040b9bb0 2024-03-19T01:23:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.575+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f040413d650 con 0x7f04040b9bb0 2024-03-19T01:23:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.575+0000 7f03fa7fc640 1 -- 172.21.15.7:0/2504411161 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f03f40f03d0 con 0x7f04040b9bb0 2024-03-19T01:23:54.577 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03e8067660 msgr2=0x7f03e8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03e8067660 0x7f03e8069b20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0400009790 tx=0x7f0400008040 comp rx=0 tx=0).stop 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040b9bb0 msgr2=0x7f0404138020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040b9bb0 0x7f0404138020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f03f401aab0 tx=0x7f03f40191a0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 shutdown_connections 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f03e8067660 0x7f03e8069b20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0404147bb0 0x7f0404143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04040b9bb0 0x7f0404138020 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 --2- 172.21.15.7:0/2504411161 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04040af4f0 0x7f040406b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 >> 172.21.15.7:0/2504411161 conn(0x7f040406e1b0 msgr2=0x7f040413c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 shutdown_connections 2024-03-19T01:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.579+0000 7f040bcb4640 1 -- 172.21.15.7:0/2504411161 wait complete. 2024-03-19T01:23:54.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T01:23:54.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.715+0000 7f2ef2fd8640 1 Processor -- start 2024-03-19T01:23:54.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.715+0000 7f2ef2fd8640 1 -- start start 2024-03-19T01:23:54.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.715+0000 7f2ef2fd8640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2eec0b47b0 0x7f2eec14a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.715+0000 7f2ef2fd8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eec14b340 con 0x7f2eec0b9ed0 2024-03-19T01:23:54.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.715+0000 7f2ef2fd8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eec0ba2b0 con 0x7f2eec14aef0 2024-03-19T01:23:54.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.715+0000 7f2ef2fd8640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f2eec0b6cc0 con 0x7f2eec0b47b0 2024-03-19T01:23:54.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.715+0000 7f2ef0d74640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2eec0b9ed0 0x7f2eec1411d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35166/0 (socket says 172.21.15.7:35166) 2024-03-19T01:23:54.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.715+0000 7f2ef0d74640 1 -- 172.21.15.7:0/2007071424 learned_addr learned my addr 172.21.15.7:0/2007071424 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:54.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eebfff640 1 --2- 172.21.15.7:0/2007071424 >> v2:172.21.15.167:3300/0 conn(0x7f2eec0b47b0 0x7f2eec14a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2320055510 0 0) 0x7f2eec14b340 con 0x7f2eec0b9ed0 2024-03-19T01:23:54.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2edc003530 con 0x7f2eec0b9ed0 2024-03-19T01:23:54.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 700034284 0 0) 0x7f2edc003530 con 0x7f2eec0b9ed0 2024-03-19T01:23:54.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 >> v2:172.21.15.167:3300/0 conn(0x7f2eec0b47b0 msgr2=0x7f2eec14a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:54.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 --2- 172.21.15.7:0/2007071424 >> v2:172.21.15.167:3300/0 conn(0x7f2eec0b47b0 0x7f2eec14a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 >> v1:172.21.15.167:6789/0 conn(0x7f2eec14aef0 legacy=0x7f2eec151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec0b52b0 con 0x7f2eec0b9ed0 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f2ee0001680 con 0x7f2eec0b9ed0 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 >> v1:172.21.15.7:6789/0 conn(0x7f2eec0b9ed0 legacy=0x7f2eec1411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2edc0039c0 0x7f2edc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2edc004810 0x7f2edc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0052f0 0x7f2edc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec0b6cc0 con 0x7f2edc0039c0 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc00c1c0 con 0x7f2edc0052f0 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2edc00c3c0 con 0x7f2edc004810 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef0d74640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2edc0039c0 0x7f2edc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eebfff640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2edc004810 0x7f2edc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef0d74640 1 -- 172.21.15.7:0/2007071424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2edc004810 msgr2=0x7f2edc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:54.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef1575640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0052f0 0x7f2edc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef0d74640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2edc004810 0x7f2edc004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef0d74640 1 -- 172.21.15.7:0/2007071424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0052f0 msgr2=0x7f2edc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:54.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef0d74640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0052f0 0x7f2edc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef0d74640 1 -- 172.21.15.7:0/2007071424 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2edc003530 con 0x7f2edc0039c0 2024-03-19T01:23:54.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef1575640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0052f0 0x7f2edc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:54.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef1575640 1 -- 172.21.15.7:0/2007071424 reap_dead start 2024-03-19T01:23:54.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2ef0d74640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2edc0039c0 0x7f2edc003de0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000c210 tx=0x7f2ee001cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc729b3c5ebf9e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0007d20 con 0x7f2edc0039c0 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ee00a9070 con 0x7f2edc0039c0 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.719+0000 7f2eeb7fe640 1 -- 172.21.15.7:0/2007071424 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0005da0 con 0x7f2edc0039c0 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2007071424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2edc0039c0 msgr2=0x7f2edc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2edc0039c0 0x7f2edc003de0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000c210 tx=0x7f2ee001cc30 comp rx=0 tx=0).stop 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2007071424 shutdown_connections 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 --2- 172.21.15.7:0/2007071424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2edc0039c0 0x7f2edc003de0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2007071424 >> 172.21.15.7:0/2007071424 conn(0x7f2eec072df0 msgr2=0x7f2eec073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2007071424 shutdown_connections 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2007071424 wait complete. 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 Processor -- start 2024-03-19T01:23:54.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- start start 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0039c0 0x7f2eec149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eec1e01c0 0x7f2eec136090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eec1e0640 0x7f2eec1365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec0b4d80 con 0x7f2eec1e01c0 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec0b4c00 con 0x7f2edc0039c0 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2eec0b7640 con 0x7f2eec1e0640 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2eebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eec1e01c0 0x7f2eec136090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2eebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eec1e01c0 0x7f2eec136090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49462/0 (socket says 172.21.15.7:49462) 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2eebfff640 1 -- 172.21.15.7:0/2876879434 learned_addr learned my addr 172.21.15.7:0/2876879434 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef1575640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eec1e0640 0x7f2eec1365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef0d74640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0039c0 0x7f2eec149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2eebfff640 1 -- 172.21.15.7:0/2876879434 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eec1e0640 msgr2=0x7f2eec1365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2eebfff640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eec1e0640 0x7f2eec1365d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2eebfff640 1 -- 172.21.15.7:0/2876879434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0039c0 msgr2=0x7f2eec149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2eebfff640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0039c0 0x7f2eec149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2eebfff640 1 -- 172.21.15.7:0/2876879434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec0bc600 con 0x7f2eec1e01c0 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef0d74640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0039c0 0x7f2eec149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2eebfff640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eec1e01c0 0x7f2eec136090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8010900 tx=0x7f2ed8010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ee97fa640 1 -- 172.21.15.7:0/2876879434 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed800f040 con 0x7f2eec1e01c0 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ee97fa640 1 -- 172.21.15.7:0/2876879434 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ed800ec20 con 0x7f2eec1e01c0 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eec0bc8c0 con 0x7f2eec1e01c0 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ee97fa640 1 -- 172.21.15.7:0/2876879434 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed801c400 con 0x7f2eec1e01c0 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eec0bd460 con 0x7f2eec1e01c0 2024-03-19T01:23:54.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.723+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eec075460 con 0x7f2eec1e01c0 2024-03-19T01:23:54.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.727+0000 7f2ee97fa640 1 -- 172.21.15.7:0/2876879434 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2ed801c5a0 con 0x7f2eec1e01c0 2024-03-19T01:23:54.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.727+0000 7f2ee97fa640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2ed0067660 0x7f2ed0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:54.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.727+0000 7f2ef0d74640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2ed0067660 0x7f2ed0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:54.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.727+0000 7f2ee97fa640 1 -- 172.21.15.7:0/2876879434 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2ed8085e70 con 0x7f2eec1e01c0 2024-03-19T01:23:54.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.727+0000 7f2ef0d74640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2ed0067660 0x7f2ed0069b20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00ab000 tx=0x7f2ee00a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:54.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.731+0000 7f2ee97fa640 1 -- 172.21.15.7:0/2876879434 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2ed8018030 con 0x7f2eec1e01c0 2024-03-19T01:23:54.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.951+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2eec140800 con 0x7f2eec1e01c0 2024-03-19T01:23:54.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.951+0000 7f2ee97fa640 1 -- 172.21.15.7:0/2876879434 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f2ed8056b80 con 0x7f2eec1e01c0 2024-03-19T01:23:54.954 INFO:teuthology.orchestra.run.smithi007.stderr:listed 0 entries 2024-03-19T01:23:54.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2ed0067660 msgr2=0x7f2ed0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2ed0067660 0x7f2ed0069b20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00ab000 tx=0x7f2ee00a7070 comp rx=0 tx=0).stop 2024-03-19T01:23:54.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eec1e01c0 msgr2=0x7f2eec136090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:54.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eec1e01c0 0x7f2eec136090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8010900 tx=0x7f2ed8010dc0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 shutdown_connections 2024-03-19T01:23:54.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2ed0067660 0x7f2ed0069b20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eec1e0640 0x7f2eec1365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eec1e01c0 0x7f2eec136090 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 --2- 172.21.15.7:0/2876879434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2edc0039c0 0x7f2eec149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:54.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 >> 172.21.15.7:0/2876879434 conn(0x7f2eec072df0 msgr2=0x7f2eec146590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:54.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 shutdown_connections 2024-03-19T01:23:54.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:54.955+0000 7f2ef2fd8640 1 -- 172.21.15.7:0/2876879434 wait complete. 2024-03-19T01:23:54.981 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:23:54.981 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:23:54.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:55.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7ec06640 1 Processor -- start 2024-03-19T01:23:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7ec06640 1 -- start start 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7ec06640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad78071b60 0x7fad780b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7ec06640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fad78145cf0 0x7fad7813ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7ec06640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad781454f0 con 0x7fad78071780 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7ec06640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fad78145860 con 0x7fad78071b60 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7ec06640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fad78146160 con 0x7fad78145cf0 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad77fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad78071b60 0x7fad780b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7c9a2640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fad78071780 0x7fad78072da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35172/0 (socket says 172.21.15.7:35172) 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7c9a2640 1 -- 172.21.15.7:0/4243101395 learned_addr learned my addr 172.21.15.7:0/4243101395 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad7d1a3640 1 --2- 172.21.15.7:0/4243101395 >> v2:172.21.15.167:3300/0 conn(0x7fad78145cf0 0x7fad7813ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad77fff640 1 -- 172.21.15.7:0/4243101395 >> v2:172.21.15.167:3300/0 conn(0x7fad78145cf0 msgr2=0x7fad7813ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:55.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad77fff640 1 --2- 172.21.15.7:0/4243101395 >> v2:172.21.15.167:3300/0 conn(0x7fad78145cf0 0x7fad7813ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad77fff640 1 -- 172.21.15.7:0/4243101395 >> v1:172.21.15.7:6789/0 conn(0x7fad78071780 legacy=0x7fad78072da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.147+0000 7fad77fff640 1 -- 172.21.15.7:0/4243101395 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad781451e0 con 0x7fad78071b60 2024-03-19T01:23:55.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad77fff640 1 --2- 172.21.15.7:0/4243101395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78071b60 0x7fad780b5460 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fad780b8910 tx=0x7fad6c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c856ba124dfa86 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:55.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad777fe640 1 -- 172.21.15.7:0/4243101395 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c0b0070 con 0x7fad78071b60 2024-03-19T01:23:55.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad777fe640 1 -- 172.21.15.7:0/4243101395 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad6c0ab600 con 0x7fad78071b60 2024-03-19T01:23:55.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad777fe640 1 -- 172.21.15.7:0/4243101395 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c0ab9d0 con 0x7fad78071b60 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- 172.21.15.7:0/4243101395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78071b60 msgr2=0x7fad780b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 --2- 172.21.15.7:0/4243101395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78071b60 0x7fad780b5460 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fad780b8910 tx=0x7fad6c01c3f0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- 172.21.15.7:0/4243101395 shutdown_connections 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 --2- 172.21.15.7:0/4243101395 >> v2:172.21.15.167:3300/0 conn(0x7fad78145cf0 0x7fad7813ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 --2- 172.21.15.7:0/4243101395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78071b60 0x7fad780b5460 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- 172.21.15.7:0/4243101395 >> 172.21.15.7:0/4243101395 conn(0x7fad7806bc10 msgr2=0x7fad780b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- 172.21.15.7:0/4243101395 shutdown_connections 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- 172.21.15.7:0/4243101395 wait complete. 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 Processor -- start 2024-03-19T01:23:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- start start 2024-03-19T01:23:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad78071780 0x7fad78144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad78071b60 0x7fad781e33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78145cf0 0x7fad780b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fad78152710 con 0x7fad78145cf0 2024-03-19T01:23:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fad78152590 con 0x7fad78071780 2024-03-19T01:23:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fad78152890 con 0x7fad78071b60 2024-03-19T01:23:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7d1a3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78145cf0 0x7fad780b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7d1a3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78145cf0 0x7fad780b3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49484/0 (socket says 172.21.15.7:49484) 2024-03-19T01:23:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7d1a3640 1 -- 172.21.15.7:0/4254715854 learned_addr learned my addr 172.21.15.7:0/4254715854 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7c9a2640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad78071780 0x7fad78144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad77fff640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad78071b60 0x7fad781e33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7d1a3640 1 -- 172.21.15.7:0/4254715854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad78071b60 msgr2=0x7fad781e33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7d1a3640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad78071b60 0x7fad781e33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7d1a3640 1 -- 172.21.15.7:0/4254715854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad78071780 msgr2=0x7fad78144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7d1a3640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad78071780 0x7fad78144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7d1a3640 1 -- 172.21.15.7:0/4254715854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7806d760 con 0x7fad78145cf0 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7c9a2640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad78071780 0x7fad78144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7d1a3640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78145cf0 0x7fad780b3ea0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad70007e90 tx=0x7fad7000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad757fa640 1 -- 172.21.15.7:0/4254715854 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70014420 con 0x7fad78145cf0 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad757fa640 1 -- 172.21.15.7:0/4254715854 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad70019070 con 0x7fad78145cf0 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad7813fb90 con 0x7fad78145cf0 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad757fa640 1 -- 172.21.15.7:0/4254715854 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70014800 con 0x7fad78145cf0 2024-03-19T01:23:55.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.151+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad7806c680 con 0x7fad78145cf0 2024-03-19T01:23:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.155+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad78075460 con 0x7fad78145cf0 2024-03-19T01:23:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.155+0000 7fad757fa640 1 -- 172.21.15.7:0/4254715854 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fad700149a0 con 0x7fad78145cf0 2024-03-19T01:23:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.155+0000 7fad757fa640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fad64067660 0x7fad64069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.155+0000 7fad757fa640 1 -- 172.21.15.7:0/4254715854 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fad70086070 con 0x7fad78145cf0 2024-03-19T01:23:55.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.155+0000 7fad7c9a2640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fad64067660 0x7fad64069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.155+0000 7fad7c9a2640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fad64067660 0x7fad64069b20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fad68000f70 tx=0x7fad68009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:55.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.159+0000 7fad757fa640 1 -- 172.21.15.7:0/4254715854 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fad70011260 con 0x7fad78145cf0 2024-03-19T01:23:55.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.327+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad78135230 con 0x7fad78145cf0 2024-03-19T01:23:55.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.327+0000 7fad757fa640 1 -- 172.21.15.7:0/4254715854 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7fad70056d80 con 0x7fad78145cf0 2024-03-19T01:23:55.330 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:55.330 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fad64067660 msgr2=0x7fad64069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fad64067660 0x7fad64069b20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fad68000f70 tx=0x7fad68009040 comp rx=0 tx=0).stop 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78145cf0 msgr2=0x7fad780b3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78145cf0 0x7fad780b3ea0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad70007e90 tx=0x7fad7000e4b0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 shutdown_connections 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fad64067660 0x7fad64069b20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fad78071b60 0x7fad781e33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad78145cf0 0x7fad780b3ea0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 --2- 172.21.15.7:0/4254715854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad78071780 0x7fad78144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 >> 172.21.15.7:0/4254715854 conn(0x7fad7806bc10 msgr2=0x7fad78067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:55.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 shutdown_connections 2024-03-19T01:23:55.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.331+0000 7fad7ec06640 1 -- 172.21.15.7:0/4254715854 wait complete. 2024-03-19T01:23:55.349 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:55.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:55.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.471+0000 7f80cffde640 1 Processor -- start 2024-03-19T01:23:55.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.471+0000 7f80cffde640 1 -- start start 2024-03-19T01:23:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80cffde640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 0x7f80c80afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80cffde640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80c80b1c50 0x7f80c8143860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80cffde640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80c80b2170 con 0x7f80c80b0ce0 2024-03-19T01:23:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80cffde640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f80c80b2450 con 0x7f80c80b1c50 2024-03-19T01:23:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80cffde640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f80c8073830 con 0x7f80c80b0900 2024-03-19T01:23:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80cd579640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f80c80b0ce0 0x7f80c8073120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35182/0 (socket says 172.21.15.7:35182) 2024-03-19T01:23:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80ce57b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80c80b1c50 0x7f80c8143860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80cd579640 1 -- 172.21.15.7:0/2730873848 learned_addr learned my addr 172.21.15.7:0/2730873848 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:55.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80cdd7a640 1 --2- 172.21.15.7:0/2730873848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 0x7f80c80afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80ce57b640 1 -- 172.21.15.7:0/2730873848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 msgr2=0x7f80c80afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:55.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80ce57b640 1 --2- 172.21.15.7:0/2730873848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 0x7f80c80afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80ce57b640 1 -- 172.21.15.7:0/2730873848 >> v1:172.21.15.7:6789/0 conn(0x7f80c80b0ce0 legacy=0x7f80c8073120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80ce57b640 1 -- 172.21.15.7:0/2730873848 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c80b1160 con 0x7f80c80b1c50 2024-03-19T01:23:55.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80ce57b640 1 --2- 172.21.15.7:0/2730873848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b1c50 0x7f80c8143860 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009d90 tx=0x7f80c001e740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32692d28e483bfdc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:55.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80ccd78640 1 -- 172.21.15.7:0/2730873848 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c00b3070 con 0x7f80c80b1c50 2024-03-19T01:23:55.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80ccd78640 1 -- 172.21.15.7:0/2730873848 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80c00ae440 con 0x7f80c80b1c50 2024-03-19T01:23:55.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.475+0000 7f80ccd78640 1 -- 172.21.15.7:0/2730873848 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c00ae810 con 0x7f80c80b1c50 2024-03-19T01:23:55.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- 172.21.15.7:0/2730873848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b1c50 msgr2=0x7f80c8143860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 --2- 172.21.15.7:0/2730873848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b1c50 0x7f80c8143860 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009d90 tx=0x7f80c001e740 comp rx=0 tx=0).stop 2024-03-19T01:23:55.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- 172.21.15.7:0/2730873848 shutdown_connections 2024-03-19T01:23:55.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 --2- 172.21.15.7:0/2730873848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b1c50 0x7f80c8143860 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 --2- 172.21.15.7:0/2730873848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 0x7f80c80afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- 172.21.15.7:0/2730873848 >> 172.21.15.7:0/2730873848 conn(0x7f80c80b9120 msgr2=0x7f80c80b9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:55.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- 172.21.15.7:0/2730873848 shutdown_connections 2024-03-19T01:23:55.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- 172.21.15.7:0/2730873848 wait complete. 2024-03-19T01:23:55.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 Processor -- start 2024-03-19T01:23:55.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- start start 2024-03-19T01:23:55.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 0x7f80c813d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b0ce0 0x7f80c80ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80c80b1c50 0x7f80c80addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f80c81511a0 con 0x7f80c80b0ce0 2024-03-19T01:23:55.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f80c8151020 con 0x7f80c80b1c50 2024-03-19T01:23:55.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f80c8151320 con 0x7f80c80b0900 2024-03-19T01:23:55.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cd579640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b0ce0 0x7f80c80ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cd579640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b0ce0 0x7f80c80ad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49506/0 (socket says 172.21.15.7:49506) 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cd579640 1 -- 172.21.15.7:0/1031590280 learned_addr learned my addr 172.21.15.7:0/1031590280 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cdd7a640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 0x7f80c813d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80ce57b640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80c80b1c50 0x7f80c80addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cd579640 1 -- 172.21.15.7:0/1031590280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 msgr2=0x7f80c813d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cd579640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 0x7f80c813d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cd579640 1 -- 172.21.15.7:0/1031590280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80c80b1c50 msgr2=0x7f80c80addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cd579640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80c80b1c50 0x7f80c80addb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cd579640 1 -- 172.21.15.7:0/1031590280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c8070800 con 0x7f80c80b0ce0 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cd579640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b0ce0 0x7f80c80ad870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f80b800c400 tx=0x7f80b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80be7fc640 1 -- 172.21.15.7:0/1031590280 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8017070 con 0x7f80c80b0ce0 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80be7fc640 1 -- 172.21.15.7:0/1031590280 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80b8005cf0 con 0x7f80c80b0ce0 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c81e0af0 con 0x7f80c80b0ce0 2024-03-19T01:23:55.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80be7fc640 1 -- 172.21.15.7:0/1031590280 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8007310 con 0x7f80c80b0ce0 2024-03-19T01:23:55.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c81e10b0 con 0x7f80c80b0ce0 2024-03-19T01:23:55.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.479+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c81e12b0 con 0x7f80c80b0ce0 2024-03-19T01:23:55.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.483+0000 7f80be7fc640 1 -- 172.21.15.7:0/1031590280 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f80b801a070 con 0x7f80c80b0ce0 2024-03-19T01:23:55.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.483+0000 7f80be7fc640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ac067660 0x7f80ac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.483+0000 7f80cdd7a640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ac067660 0x7f80ac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.483+0000 7f80be7fc640 1 -- 172.21.15.7:0/1031590280 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f80b8084c30 con 0x7f80c80b0ce0 2024-03-19T01:23:55.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.483+0000 7f80cdd7a640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ac067660 0x7f80ac069b20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f80c400b9d0 tx=0x7f80c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:55.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.487+0000 7f80be7fc640 1 -- 172.21.15.7:0/1031590280 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f80b8055940 con 0x7f80c80b0ce0 2024-03-19T01:23:55.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.647+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f80c80b1630 con 0x7f80c80b0ce0 2024-03-19T01:23:55.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.647+0000 7f80be7fc640 1 -- 172.21.15.7:0/1031590280 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f80b8055240 con 0x7f80c80b0ce0 2024-03-19T01:23:55.649 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:55.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ac067660 msgr2=0x7f80ac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ac067660 0x7f80ac069b20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f80c400b9d0 tx=0x7f80c4008040 comp rx=0 tx=0).stop 2024-03-19T01:23:55.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b0ce0 msgr2=0x7f80c80ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b0ce0 0x7f80c80ad870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f80b800c400 tx=0x7f80b800c8c0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 shutdown_connections 2024-03-19T01:23:55.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f80ac067660 0x7f80ac069b20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80c80b1c50 0x7f80c80addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c80b0ce0 0x7f80c80ad870 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 --2- 172.21.15.7:0/1031590280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c80b0900 0x7f80c813d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 >> 172.21.15.7:0/1031590280 conn(0x7f80c80b9120 msgr2=0x7f80c80b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:55.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 shutdown_connections 2024-03-19T01:23:55.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.651+0000 7f80cffde640 1 -- 172.21.15.7:0/1031590280 wait complete. 2024-03-19T01:23:55.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b12739640 1 Processor -- start 2024-03-19T01:23:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b12739640 1 -- start start 2024-03-19T01:23:55.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b12739640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b0c0714a0 0x7f7b0c072110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b12739640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b0c0b0c70 con 0x7f7b0c144b00 2024-03-19T01:23:55.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b12739640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b0c071880 con 0x7f7b0c13e030 2024-03-19T01:23:55.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b12739640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0c072770 con 0x7f7b0c0714a0 2024-03-19T01:23:55.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b10cd6640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7b0c144b00 0x7f7b0c14e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35184/0 (socket says 172.21.15.7:35184) 2024-03-19T01:23:55.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b10cd6640 1 -- 172.21.15.7:0/4227349881 learned_addr learned my addr 172.21.15.7:0/4227349881 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:55.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0bfff640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b0c0714a0 0x7f7b0c072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3757753255 0 0) 0x7f7b0c0b0c70 con 0x7f7b0c144b00 2024-03-19T01:23:55.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7af4003530 con 0x7f7b0c144b00 2024-03-19T01:23:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2808220697 0 0) 0x7f7af4003530 con 0x7f7b0c144b00 2024-03-19T01:23:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b0c0714a0 msgr2=0x7f7b0c072110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b0c0714a0 0x7f7b0c072110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 >> v1:172.21.15.167:6789/0 conn(0x7f7b0c13e030 legacy=0x7f7b0c1443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0c070a60 con 0x7f7b0c144b00 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f7b04001a00 con 0x7f7b0c144b00 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 >> v1:172.21.15.7:6789/0 conn(0x7f7b0c144b00 legacy=0x7f7b0c14e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af40039f0 0x7f7af4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7af4004810 0x7f7af4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b0c0714a0 0x7f7af4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c072770 con 0x7f7af40039f0 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7af400bb00 con 0x7f7af4004810 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7af400bd00 con 0x7f7b0c0714a0 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0bfff640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af40039f0 0x7f7af4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0b7fe640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7af4004810 0x7f7af4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b10cd6640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b0c0714a0 0x7f7af4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0bfff640 1 -- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b0c0714a0 msgr2=0x7f7af4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0bfff640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b0c0714a0 0x7f7af4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0bfff640 1 -- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7af4004810 msgr2=0x7f7af4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0bfff640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7af4004810 0x7f7af4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0bfff640 1 -- 172.21.15.7:0/4227349881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af4003530 con 0x7f7af40039f0 2024-03-19T01:23:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.791+0000 7f7b0b7fe640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7af4004810 0x7f7af4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b0bfff640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af40039f0 0x7f7af4003dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00bd60 tx=0x7f7afc01f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0d93372e447092 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc0b3070 con 0x7f7af40039f0 2024-03-19T01:23:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7afc0ae600 con 0x7f7af40039f0 2024-03-19T01:23:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b0affd640 1 -- 172.21.15.7:0/4227349881 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc00b930 con 0x7f7af40039f0 2024-03-19T01:23:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b12739640 1 -- 172.21.15.7:0/4227349881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af40039f0 msgr2=0x7f7af4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b12739640 1 --2- 172.21.15.7:0/4227349881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af40039f0 0x7f7af4003dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00bd60 tx=0x7f7afc01f370 comp rx=0 tx=0).stop 2024-03-19T01:23:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b10cd6640 1 -- 172.21.15.7:0/4227349881 reap_dead start 2024-03-19T01:23:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b12739640 1 -- 172.21.15.7:0/4227349881 shutdown_connections 2024-03-19T01:23:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b12739640 1 -- 172.21.15.7:0/4227349881 >> 172.21.15.7:0/4227349881 conn(0x7f7b0c0b7980 msgr2=0x7f7b0c0b7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b12739640 1 -- 172.21.15.7:0/4227349881 shutdown_connections 2024-03-19T01:23:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.795+0000 7f7b12739640 1 -- 172.21.15.7:0/4227349881 wait complete. 2024-03-19T01:23:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 Processor -- start 2024-03-19T01:23:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 -- start start 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7af40039f0 0x7f7b0c1df1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b0c1452a0 0x7f7b0c1df990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c135930 0x7f7b0c135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c06bad0 con 0x7f7b0c135930 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c13e840 con 0x7f7b0c1452a0 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0c06bc50 con 0x7f7af40039f0 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b10cd6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c135930 0x7f7b0c135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b10cd6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c135930 0x7f7b0c135d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49536/0 (socket says 172.21.15.7:49536) 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b10cd6640 1 -- 172.21.15.7:0/2809095777 learned_addr learned my addr 172.21.15.7:0/2809095777 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b0bfff640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7af40039f0 0x7f7b0c1df1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b0b7fe640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b0c1452a0 0x7f7b0c1df990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b10cd6640 1 -- 172.21.15.7:0/2809095777 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7af40039f0 msgr2=0x7f7b0c1df1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b10cd6640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7af40039f0 0x7f7b0c1df1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b10cd6640 1 -- 172.21.15.7:0/2809095777 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b0c1452a0 msgr2=0x7f7b0c1df990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b10cd6640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b0c1452a0 0x7f7b0c1df990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b10cd6640 1 -- 172.21.15.7:0/2809095777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0c147f00 con 0x7f7b0c135930 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b0b7fe640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b0c1452a0 0x7f7b0c1df990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b10cd6640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c135930 0x7f7b0c135d90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c0680d0 tx=0x7f7b0400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b08ff9640 1 -- 172.21.15.7:0/2809095777 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0400ebc0 con 0x7f7b0c135930 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b08ff9640 1 -- 172.21.15.7:0/2809095777 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b0400ed60 con 0x7f7b0c135930 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0c148790 con 0x7f7b0c135930 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b08ff9640 1 -- 172.21.15.7:0/2809095777 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b04014670 con 0x7f7b0c135930 2024-03-19T01:23:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0c147c20 con 0x7f7b0c135930 2024-03-19T01:23:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.799+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b0c146da0 con 0x7f7b0c135930 2024-03-19T01:23:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.803+0000 7f7b08ff9640 1 -- 172.21.15.7:0/2809095777 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7b04014810 con 0x7f7b0c135930 2024-03-19T01:23:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.803+0000 7f7b08ff9640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7af8067660 0x7f7af8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.803+0000 7f7b08ff9640 1 -- 172.21.15.7:0/2809095777 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7b040855a0 con 0x7f7b0c135930 2024-03-19T01:23:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.803+0000 7f7b0bfff640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7af8067660 0x7f7af8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.803+0000 7f7b0bfff640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7af8067660 0x7f7af8069b20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0b0000 tx=0x7f7afc0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.807+0000 7f7b08ff9640 1 -- 172.21.15.7:0/2809095777 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7b0c146da0 con 0x7f7b0c135930 2024-03-19T01:23:55.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.963+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7b0c075570 con 0x7f7b0c135930 2024-03-19T01:23:55.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b08ff9640 1 -- 172.21.15.7:0/2809095777 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f7b0c147f00 con 0x7f7b0c135930 2024-03-19T01:23:55.968 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7af8067660 msgr2=0x7f7af8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7af8067660 0x7f7af8069b20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0b0000 tx=0x7f7afc0046d0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c135930 msgr2=0x7f7b0c135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c135930 0x7f7b0c135d90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c0680d0 tx=0x7f7b0400e530 comp rx=0 tx=0).stop 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 shutdown_connections 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7af8067660 0x7f7af8069b20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c135930 0x7f7b0c135d90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b0c1452a0 0x7f7b0c1df990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 --2- 172.21.15.7:0/2809095777 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7af40039f0 0x7f7b0c1df1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 >> 172.21.15.7:0/2809095777 conn(0x7f7b0c0b7980 msgr2=0x7f7b0c0b8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 shutdown_connections 2024-03-19T01:23:55.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:55.967+0000 7f7b12739640 1 -- 172.21.15.7:0/2809095777 wait complete. 2024-03-19T01:23:55.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:56.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84f2745640 1 Processor -- start 2024-03-19T01:23:56.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84f2745640 1 -- start start 2024-03-19T01:23:56.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84f2745640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f84ec0b6b60 0x7f84ec148790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84f2745640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f84ec148cd0 0x7f84ec0b4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84f2745640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84ec0b5020 con 0x7f84ec0b47b0 2024-03-19T01:23:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84f2745640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f84ec0b7dd0 con 0x7f84ec148cd0 2024-03-19T01:23:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84f2745640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f84ec0b5b20 con 0x7f84ec0b6b60 2024-03-19T01:23:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eb7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f84ec148cd0 0x7f84ec0b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84f0ce2640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f84ec0b47b0 0x7f84ec071450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35186/0 (socket says 172.21.15.7:35186) 2024-03-19T01:23:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84f0ce2640 1 -- 172.21.15.7:0/974676166 learned_addr learned my addr 172.21.15.7:0/974676166 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84ebfff640 1 --2- 172.21.15.7:0/974676166 >> v2:172.21.15.167:3300/0 conn(0x7f84ec0b6b60 0x7f84ec148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eb7fe640 1 -- 172.21.15.7:0/974676166 >> v2:172.21.15.167:3300/0 conn(0x7f84ec0b6b60 msgr2=0x7f84ec148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:56.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eb7fe640 1 --2- 172.21.15.7:0/974676166 >> v2:172.21.15.167:3300/0 conn(0x7f84ec0b6b60 0x7f84ec148790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eb7fe640 1 -- 172.21.15.7:0/974676166 >> v1:172.21.15.7:6789/0 conn(0x7f84ec0b47b0 legacy=0x7f84ec071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eaffd640 1 -- 172.21.15.7:0/974676166 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1791517362 0 0) 0x7f84ec0b5020 con 0x7f84ec0b47b0 2024-03-19T01:23:56.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eb7fe640 1 -- 172.21.15.7:0/974676166 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec0b5d70 con 0x7f84ec148cd0 2024-03-19T01:23:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eb7fe640 1 --2- 172.21.15.7:0/974676166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec148cd0 0x7f84ec0b4270 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0b7630 tx=0x7f84dc01c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d71c5385ae4635 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eaffd640 1 -- 172.21.15.7:0/974676166 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0a6710 con 0x7f84ec148cd0 2024-03-19T01:23:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eaffd640 1 -- 172.21.15.7:0/974676166 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84dc0a68b0 con 0x7f84ec148cd0 2024-03-19T01:23:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.107+0000 7f84eaffd640 1 -- 172.21.15.7:0/974676166 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0a6c80 con 0x7f84ec148cd0 2024-03-19T01:23:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 -- 172.21.15.7:0/974676166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec148cd0 msgr2=0x7f84ec0b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 --2- 172.21.15.7:0/974676166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec148cd0 0x7f84ec0b4270 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0b7630 tx=0x7f84dc01c810 comp rx=0 tx=0).stop 2024-03-19T01:23:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 -- 172.21.15.7:0/974676166 shutdown_connections 2024-03-19T01:23:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 --2- 172.21.15.7:0/974676166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec148cd0 0x7f84ec0b4270 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 --2- 172.21.15.7:0/974676166 >> v2:172.21.15.167:3300/0 conn(0x7f84ec0b6b60 0x7f84ec148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 -- 172.21.15.7:0/974676166 >> 172.21.15.7:0/974676166 conn(0x7f84ec06b2c0 msgr2=0x7f84ec06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 -- 172.21.15.7:0/974676166 shutdown_connections 2024-03-19T01:23:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 -- 172.21.15.7:0/974676166 wait complete. 2024-03-19T01:23:56.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 Processor -- start 2024-03-19T01:23:56.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 -- start start 2024-03-19T01:23:56.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84ec0b47b0 0x7f84ec13e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec0b6b60 0x7f84ec1ddad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84ec148cd0 0x7f84ec1de010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec0b6480 con 0x7f84ec0b6b60 2024-03-19T01:23:56.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec0b6300 con 0x7f84ec0b47b0 2024-03-19T01:23:56.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f2745640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f84ec072ad0 con 0x7f84ec148cd0 2024-03-19T01:23:56.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84eb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec0b6b60 0x7f84ec1ddad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84eb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec0b6b60 0x7f84ec1ddad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49560/0 (socket says 172.21.15.7:49560) 2024-03-19T01:23:56.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84eb7fe640 1 -- 172.21.15.7:0/46692580 learned_addr learned my addr 172.21.15.7:0/46692580 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:56.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84f0ce2640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84ec148cd0 0x7f84ec1de010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84eb7fe640 1 -- 172.21.15.7:0/46692580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84ec148cd0 msgr2=0x7f84ec1de010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84eb7fe640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84ec148cd0 0x7f84ec1de010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84ebfff640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84ec0b47b0 0x7f84ec13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84eb7fe640 1 -- 172.21.15.7:0/46692580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84ec0b47b0 msgr2=0x7f84ec13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84eb7fe640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84ec0b47b0 0x7f84ec13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84eb7fe640 1 -- 172.21.15.7:0/46692580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec141bb0 con 0x7f84ec0b6b60 2024-03-19T01:23:56.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.111+0000 7f84ebfff640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84ec0b47b0 0x7f84ec13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:56.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84eb7fe640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec0b6b60 0x7f84ec1ddad0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0b0040 tx=0x7f84dc0195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:56.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84e8ff9640 1 -- 172.21.15.7:0/46692580 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0be070 con 0x7f84ec0b6b60 2024-03-19T01:23:56.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84e8ff9640 1 -- 172.21.15.7:0/46692580 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84dc0ae8d0 con 0x7f84ec0b6b60 2024-03-19T01:23:56.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ec141e50 con 0x7f84ec0b6b60 2024-03-19T01:23:56.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84e8ff9640 1 -- 172.21.15.7:0/46692580 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0aecc0 con 0x7f84ec0b6b60 2024-03-19T01:23:56.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ec14a840 con 0x7f84ec0b6b60 2024-03-19T01:23:56.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84ec075570 con 0x7f84ec0b6b60 2024-03-19T01:23:56.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84e8ff9640 1 -- 172.21.15.7:0/46692580 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f84dc0c1070 con 0x7f84ec0b6b60 2024-03-19T01:23:56.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84e8ff9640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f84d8067660 0x7f84d8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84ebfff640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f84d8067660 0x7f84d8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84e8ff9640 1 -- 172.21.15.7:0/46692580 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f84dc120fe0 con 0x7f84ec0b6b60 2024-03-19T01:23:56.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.115+0000 7f84ebfff640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f84d8067660 0x7f84d8069b20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f84e0006fd0 tx=0x7f84e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:56.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.119+0000 7f84e8ff9640 1 -- 172.21.15.7:0/46692580 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f84dc0c8030 con 0x7f84ec0b6b60 2024-03-19T01:23:56.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.275+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f84ec0b2510 con 0x7f84ec0b6b60 2024-03-19T01:23:56.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.275+0000 7f84e8ff9640 1 -- 172.21.15.7:0/46692580 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f84dc016520 con 0x7f84ec0b6b60 2024-03-19T01:23:56.280 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:56.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f84d8067660 msgr2=0x7f84d8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f84d8067660 0x7f84d8069b20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f84e0006fd0 tx=0x7f84e0008040 comp rx=0 tx=0).stop 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec0b6b60 msgr2=0x7f84ec1ddad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec0b6b60 0x7f84ec1ddad0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0b0040 tx=0x7f84dc0195b0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 shutdown_connections 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f84d8067660 0x7f84d8069b20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84ec148cd0 0x7f84ec1de010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec0b6b60 0x7f84ec1ddad0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 --2- 172.21.15.7:0/46692580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84ec0b47b0 0x7f84ec13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 >> 172.21.15.7:0/46692580 conn(0x7f84ec06b2c0 msgr2=0x7f84ec0bb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 shutdown_connections 2024-03-19T01:23:56.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.279+0000 7f84f2745640 1 -- 172.21.15.7:0/46692580 wait complete. 2024-03-19T01:23:56.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T01:23:56.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbeabf6640 1 Processor -- start 2024-03-19T01:23:56.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbeabf6640 1 -- start start 2024-03-19T01:23:56.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbeabf6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbe4074390 0x7fdbe4148b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbeabf6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe4070d60 con 0x7fdbe40b4ec0 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbeabf6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe4074770 con 0x7fdbe413f660 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbeabf6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe4070fd0 con 0x7fdbe4074390 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdbe40b4ec0 0x7fdbe413ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35202/0 (socket says 172.21.15.7:35202) 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe3fff640 1 -- 172.21.15.7:0/3545377720 learned_addr learned my addr 172.21.15.7:0/3545377720 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe8992640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbe4074390 0x7fdbe4148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3878149561 0 0) 0x7fdbe4070d60 con 0x7fdbe40b4ec0 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbd4003530 con 0x7fdbe40b4ec0 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2965475508 0 0) 0x7fdbd4003530 con 0x7fdbe40b4ec0 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbe4074390 msgr2=0x7fdbe4148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbe4074390 0x7fdbe4148b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 >> v1:172.21.15.167:6789/0 conn(0x7fdbe413f660 legacy=0x7fdbe414faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe406fcc0 con 0x7fdbe40b4ec0 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fdbd8001680 con 0x7fdbe40b4ec0 2024-03-19T01:23:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 >> v1:172.21.15.7:6789/0 conn(0x7fdbe40b4ec0 legacy=0x7fdbe413ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd40039f0 0x7fdbd4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbd4004810 0x7fdbd4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbe4074390 0x7fdbd4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe4070fd0 con 0x7fdbd40039f0 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd400bb00 con 0x7fdbd4004810 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdbd400bd00 con 0x7fdbe4074390 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe8992640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd40039f0 0x7fdbd4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe3fff640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbd4004810 0x7fdbd4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe8992640 1 -- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbe4074390 msgr2=0x7fdbd4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe9193640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbe4074390 0x7fdbd4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe8992640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbe4074390 0x7fdbd4005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe8992640 1 -- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbd4004810 msgr2=0x7fdbd4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe8992640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbd4004810 0x7fdbd4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe8992640 1 -- 172.21.15.7:0/3545377720 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd4003530 con 0x7fdbd40039f0 2024-03-19T01:23:56.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.415+0000 7fdbe3fff640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbd4004810 0x7fdbd4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe8992640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd40039f0 0x7fdbd4003e10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009ae0 tx=0x7fdbd001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5846abc5ae8696f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00096b0 con 0x7fdbd40039f0 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd00a6450 con 0x7fdbd40039f0 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe37fe640 1 -- 172.21.15.7:0/3545377720 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00a6840 con 0x7fdbd40039f0 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 -- 172.21.15.7:0/3545377720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd40039f0 msgr2=0x7fdbd4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 --2- 172.21.15.7:0/3545377720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd40039f0 0x7fdbd4003e10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009ae0 tx=0x7fdbd001ca70 comp rx=0 tx=0).stop 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe9193640 1 -- 172.21.15.7:0/3545377720 reap_dead start 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 -- 172.21.15.7:0/3545377720 shutdown_connections 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 -- 172.21.15.7:0/3545377720 >> 172.21.15.7:0/3545377720 conn(0x7fdbe40b5bf0 msgr2=0x7fdbe40b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 -- 172.21.15.7:0/3545377720 shutdown_connections 2024-03-19T01:23:56.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 -- 172.21.15.7:0/3545377720 wait complete. 2024-03-19T01:23:56.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 Processor -- start 2024-03-19T01:23:56.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 -- start start 2024-03-19T01:23:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbd40039f0 0x7fdbe413a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe413b530 0x7fdbe4143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbe4143880 0x7fdbe4138250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe41515a0 con 0x7fdbe413b530 2024-03-19T01:23:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe4151420 con 0x7fdbe4143880 2024-03-19T01:23:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbeabf6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe4151720 con 0x7fdbd40039f0 2024-03-19T01:23:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe413b530 0x7fdbe4143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe413b530 0x7fdbe4143340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49590/0 (socket says 172.21.15.7:49590) 2024-03-19T01:23:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe3fff640 1 -- 172.21.15.7:0/861722932 learned_addr learned my addr 172.21.15.7:0/861722932 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe9193640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbe4143880 0x7fdbe4138250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe8992640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbd40039f0 0x7fdbe413a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.419+0000 7fdbe3fff640 1 -- 172.21.15.7:0/861722932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbd40039f0 msgr2=0x7fdbe413a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe3fff640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbd40039f0 0x7fdbe413a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe3fff640 1 -- 172.21.15.7:0/861722932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbe4143880 msgr2=0x7fdbe4138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe3fff640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbe4143880 0x7fdbe4138250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe3fff640 1 -- 172.21.15.7:0/861722932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe4143d00 con 0x7fdbe413b530 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe8992640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbd40039f0 0x7fdbe413a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe3fff640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe413b530 0x7fdbe4143340 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8007fd0 tx=0x7fdbd800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe17fa640 1 -- 172.21.15.7:0/861722932 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8019070 con 0x7fdbe413b530 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe17fa640 1 -- 172.21.15.7:0/861722932 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd80074e0 con 0x7fdbe413b530 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe17fa640 1 -- 172.21.15.7:0/861722932 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8004a10 con 0x7fdbe413b530 2024-03-19T01:23:56.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe413ae10 con 0x7fdbe413b530 2024-03-19T01:23:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe4145200 con 0x7fdbe413b530 2024-03-19T01:23:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbe4075460 con 0x7fdbe413b530 2024-03-19T01:23:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe17fa640 1 -- 172.21.15.7:0/861722932 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdbd8004bb0 con 0x7fdbe413b530 2024-03-19T01:23:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe17fa640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdbc8067660 0x7fdbc8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe17fa640 1 -- 172.21.15.7:0/861722932 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdbd80852f0 con 0x7fdbe413b530 2024-03-19T01:23:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe8992640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdbc8067660 0x7fdbc8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.423+0000 7fdbe8992640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdbc8067660 0x7fdbc8069b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00ad000 tx=0x7fdbd00046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.427+0000 7fdbe17fa640 1 -- 172.21.15.7:0/861722932 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdbe4143d00 con 0x7fdbe413b530 2024-03-19T01:23:56.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdbe40b7860 con 0x7fdbe413b530 2024-03-19T01:23:56.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbe17fa640 1 -- 172.21.15.7:0/861722932 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fdbd8056000 con 0x7fdbe413b530 2024-03-19T01:23:56.596 INFO:teuthology.orchestra.run.smithi007.stderr:listed 0 entries 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdbc8067660 msgr2=0x7fdbc8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdbc8067660 0x7fdbc8069b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00ad000 tx=0x7fdbd00046d0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe413b530 msgr2=0x7fdbe4143340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe413b530 0x7fdbe4143340 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8007fd0 tx=0x7fdbd800dd40 comp rx=0 tx=0).stop 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 shutdown_connections 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdbc8067660 0x7fdbc8069b20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbe4143880 0x7fdbe4138250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe413b530 0x7fdbe4143340 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 --2- 172.21.15.7:0/861722932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbd40039f0 0x7fdbe413a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 >> 172.21.15.7:0/861722932 conn(0x7fdbe40b5bf0 msgr2=0x7fdbe40b8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 shutdown_connections 2024-03-19T01:23:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.595+0000 7fdbeabf6640 1 -- 172.21.15.7:0/861722932 wait complete. 2024-03-19T01:23:56.613 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi007.front.sepia.ceph.com... 2024-03-19T01:23:56.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:56.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-19T01:23:56.637 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:23:56.638 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:23:56.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:23:56.638 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-19T01:23:56.693 INFO:tasks.cephfs.cephfs_test_case:[] 2024-03-19T01:23:56.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:56.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a99824640 1 Processor -- start 2024-03-19T01:23:56.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a99824640 1 -- start start 2024-03-19T01:23:56.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a99824640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9413f460 0x7f6a94074130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a99824640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f6a9406fe80 0x7f6a94148990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a99824640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6a94148ed0 0x7f6a940b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a99824640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f6a940b3550 con 0x7f6a94148ed0 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a99824640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f6a94071160 con 0x7f6a9406fe80 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a99824640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6a94149340 con 0x7f6a9413f460 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a937fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6a94148ed0 0x7f6a940b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a937fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6a94148ed0 0x7f6a940b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49602/0 (socket says 172.21.15.7:49602) 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a937fe640 1 -- 172.21.15.7:0/1674940698 learned_addr learned my addr 172.21.15.7:0/1674940698 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a92ffd640 1 --2- 172.21.15.7:0/1674940698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9413f460 0x7f6a94074130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a927fc640 1 --2- 172.21.15.7:0/1674940698 >> v2:172.21.15.167:3300/0 conn(0x7f6a9406fe80 0x7f6a94148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a937fe640 1 -- 172.21.15.7:0/1674940698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9413f460 msgr2=0x7f6a94074130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a937fe640 1 --2- 172.21.15.7:0/1674940698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9413f460 0x7f6a94074130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a937fe640 1 -- 172.21.15.7:0/1674940698 >> v2:172.21.15.167:3300/0 conn(0x7f6a9406fe80 msgr2=0x7f6a94148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a937fe640 1 --2- 172.21.15.7:0/1674940698 >> v2:172.21.15.167:3300/0 conn(0x7f6a9406fe80 0x7f6a94148990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.863+0000 7f6a937fe640 1 -- 172.21.15.7:0/1674940698 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a940b2f00 con 0x7f6a94148ed0 2024-03-19T01:23:56.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a937fe640 1 --2- 172.21.15.7:0/1674940698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a94148ed0 0x7f6a940b3b20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6a940695f0 tx=0x7f6a8c01c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e44ba138b1676e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:56.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a91ffb640 1 -- 172.21.15.7:0/1674940698 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c0a9430 con 0x7f6a94148ed0 2024-03-19T01:23:56.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a91ffb640 1 -- 172.21.15.7:0/1674940698 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a8c0ae070 con 0x7f6a94148ed0 2024-03-19T01:23:56.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a91ffb640 1 -- 172.21.15.7:0/1674940698 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c0a97f0 con 0x7f6a94148ed0 2024-03-19T01:23:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 -- 172.21.15.7:0/1674940698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a94148ed0 msgr2=0x7f6a940b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 --2- 172.21.15.7:0/1674940698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a94148ed0 0x7f6a940b3b20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6a940695f0 tx=0x7f6a8c01c6a0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 -- 172.21.15.7:0/1674940698 shutdown_connections 2024-03-19T01:23:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 --2- 172.21.15.7:0/1674940698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a94148ed0 0x7f6a940b3b20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 --2- 172.21.15.7:0/1674940698 >> v2:172.21.15.167:3300/0 conn(0x7f6a9406fe80 0x7f6a94148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 --2- 172.21.15.7:0/1674940698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9413f460 0x7f6a94074130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 -- 172.21.15.7:0/1674940698 >> 172.21.15.7:0/1674940698 conn(0x7f6a940b5c20 msgr2=0x7f6a940b6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 -- 172.21.15.7:0/1674940698 shutdown_connections 2024-03-19T01:23:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 -- 172.21.15.7:0/1674940698 wait complete. 2024-03-19T01:23:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 Processor -- start 2024-03-19T01:23:56.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.867+0000 7f6a99824640 1 -- start start 2024-03-19T01:23:56.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a99824640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9406fe80 0x7f6a9413da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a99824640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a9413f460 0x7f6a94147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a99824640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a94148ed0 0x7f6a94131360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a99824640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6a940b4eb0 con 0x7f6a9413f460 2024-03-19T01:23:56.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a99824640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6a940b5750 con 0x7f6a94148ed0 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a99824640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6a940b5030 con 0x7f6a9406fe80 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a927fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a9413f460 0x7f6a94147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a927fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a9413f460 0x7f6a94147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49618/0 (socket says 172.21.15.7:49618) 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a927fc640 1 -- 172.21.15.7:0/507304182 learned_addr learned my addr 172.21.15.7:0/507304182 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a937fe640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a94148ed0 0x7f6a94131360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a92ffd640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9406fe80 0x7f6a9413da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a927fc640 1 -- 172.21.15.7:0/507304182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9406fe80 msgr2=0x7f6a9413da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a927fc640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9406fe80 0x7f6a9413da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a927fc640 1 -- 172.21.15.7:0/507304182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a94148ed0 msgr2=0x7f6a94131360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a927fc640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a94148ed0 0x7f6a94131360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a927fc640 1 -- 172.21.15.7:0/507304182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9414f9c0 con 0x7f6a9413f460 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a927fc640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a9413f460 0x7f6a94147890 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800d7d0 tx=0x7f6a8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:56.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a7bfff640 1 -- 172.21.15.7:0/507304182 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8800b6c0 con 0x7f6a9413f460 2024-03-19T01:23:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a7bfff640 1 -- 172.21.15.7:0/507304182 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a88019070 con 0x7f6a9413f460 2024-03-19T01:23:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a941473b0 con 0x7f6a9413f460 2024-03-19T01:23:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a7bfff640 1 -- 172.21.15.7:0/507304182 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8800ba80 con 0x7f6a9413f460 2024-03-19T01:23:56.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a94146a80 con 0x7f6a9413f460 2024-03-19T01:23:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a94075460 con 0x7f6a9413f460 2024-03-19T01:23:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a7bfff640 1 -- 172.21.15.7:0/507304182 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6a8800bc20 con 0x7f6a9413f460 2024-03-19T01:23:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a7bfff640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6a80067660 0x7f6a80069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a7bfff640 1 -- 172.21.15.7:0/507304182 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6a88085cd0 con 0x7f6a9413f460 2024-03-19T01:23:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.871+0000 7f6a92ffd640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6a80067660 0x7f6a80069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:56.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.875+0000 7f6a92ffd640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6a80067660 0x7f6a80069b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009a50 tx=0x7f6a84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:56.879+0000 7f6a7bfff640 1 -- 172.21.15.7:0/507304182 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6a88056950 con 0x7f6a9413f460 2024-03-19T01:23:57.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a940b3e40 con 0x7f6a9413f460 2024-03-19T01:23:57.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a7bfff640 1 -- 172.21.15.7:0/507304182 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3524 (secure 0 0 0) 0x7f6a88056250 con 0x7f6a9413f460 2024-03-19T01:23:57.069 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:57.069 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4208,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/656985991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":656985991},{"type":"v1","addr":"172.21.15.167:6833","nonce":656985991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:23:26.868603+0000","modified":"2024-03-19T01:23:29.918429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-03-19T01:23:57.069 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6a80067660 msgr2=0x7f6a80069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6a80067660 0x7f6a80069b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009a50 tx=0x7f6a84008040 comp rx=0 tx=0).stop 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a9413f460 msgr2=0x7f6a94147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a9413f460 0x7f6a94147890 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800d7d0 tx=0x7f6a8800dc90 comp rx=0 tx=0).stop 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 shutdown_connections 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6a80067660 0x7f6a80069b20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a94148ed0 0x7f6a94131360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a9413f460 0x7f6a94147890 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 --2- 172.21.15.7:0/507304182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a9406fe80 0x7f6a9413da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 >> 172.21.15.7:0/507304182 conn(0x7f6a940b5c20 msgr2=0x7f6a940b97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 shutdown_connections 2024-03-19T01:23:57.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.067+0000 7f6a99824640 1 -- 172.21.15.7:0/507304182 wait complete. 2024-03-19T01:23:57.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:57.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3c817640 1 Processor -- start 2024-03-19T01:23:57.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3c817640 1 -- start start 2024-03-19T01:23:57.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3c817640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d380b9ed0 0x7f6d381411d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3c817640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f6d380b47b0 0x7f6d3814a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3c817640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d38141710 0x7f6d3814aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3c817640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f6d380b6cc0 con 0x7f6d380b9ed0 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3c817640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f6d380ba2b0 con 0x7f6d380b47b0 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3c817640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6d380b52b0 con 0x7f6d38141710 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d380b9ed0 0x7f6d381411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d380b9ed0 0x7f6d381411d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49626/0 (socket says 172.21.15.7:49626) 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3659c640 1 -- 172.21.15.7:0/1632481670 learned_addr learned my addr 172.21.15.7:0/1632481670 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d35d9b640 1 --2- 172.21.15.7:0/1632481670 >> v2:172.21.15.167:3300/0 conn(0x7f6d380b47b0 0x7f6d3814a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d36d9d640 1 --2- 172.21.15.7:0/1632481670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d38141710 0x7f6d3814aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3659c640 1 -- 172.21.15.7:0/1632481670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d38141710 msgr2=0x7f6d3814aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3659c640 1 --2- 172.21.15.7:0/1632481670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d38141710 0x7f6d3814aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3659c640 1 -- 172.21.15.7:0/1632481670 >> v2:172.21.15.167:3300/0 conn(0x7f6d380b47b0 msgr2=0x7f6d3814a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3659c640 1 --2- 172.21.15.7:0/1632481670 >> v2:172.21.15.167:3300/0 conn(0x7f6d380b47b0 0x7f6d3814a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d3659c640 1 -- 172.21.15.7:0/1632481670 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d380b5590 con 0x7f6d380b9ed0 2024-03-19T01:23:57.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d35d9b640 1 --2- 172.21.15.7:0/1632481670 >> v2:172.21.15.167:3300/0 conn(0x7f6d380b47b0 0x7f6d3814a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:57.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.215+0000 7f6d36d9d640 1 --2- 172.21.15.7:0/1632481670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d38141710 0x7f6d3814aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3659c640 1 --2- 172.21.15.7:0/1632481670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 0x7f6d381411d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6d380b6850 tx=0x7f6d2801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7639a57bac912dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3559a640 1 -- 172.21.15.7:0/1632481670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d280b1070 con 0x7f6d380b9ed0 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3559a640 1 -- 172.21.15.7:0/1632481670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d280a8050 con 0x7f6d380b9ed0 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3559a640 1 -- 172.21.15.7:0/1632481670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d280ac860 con 0x7f6d380b9ed0 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 -- 172.21.15.7:0/1632481670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 msgr2=0x7f6d381411d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 --2- 172.21.15.7:0/1632481670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 0x7f6d381411d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6d380b6850 tx=0x7f6d2801c400 comp rx=0 tx=0).stop 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 -- 172.21.15.7:0/1632481670 shutdown_connections 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 --2- 172.21.15.7:0/1632481670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d38141710 0x7f6d3814aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 --2- 172.21.15.7:0/1632481670 >> v2:172.21.15.167:3300/0 conn(0x7f6d380b47b0 0x7f6d3814a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 --2- 172.21.15.7:0/1632481670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 0x7f6d381411d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 -- 172.21.15.7:0/1632481670 >> 172.21.15.7:0/1632481670 conn(0x7f6d38072df0 msgr2=0x7f6d38073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 -- 172.21.15.7:0/1632481670 shutdown_connections 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 -- 172.21.15.7:0/1632481670 wait complete. 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 Processor -- start 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 -- start start 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d380b47b0 0x7f6d381446a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 0x7f6d3813b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d38141710 0x7f6d3813b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6d380b7fd0 con 0x7f6d380b9ed0 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6d380b7e50 con 0x7f6d38141710 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3c817640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6d380b8150 con 0x7f6d380b47b0 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d35d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 0x7f6d3813b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d35d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 0x7f6d3813b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49640/0 (socket says 172.21.15.7:49640) 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d35d9b640 1 -- 172.21.15.7:0/3707899534 learned_addr learned my addr 172.21.15.7:0/3707899534 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:57.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d3659c640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d380b47b0 0x7f6d381446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d36d9d640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d38141710 0x7f6d3813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d35d9b640 1 -- 172.21.15.7:0/3707899534 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d380b47b0 msgr2=0x7f6d381446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d35d9b640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d380b47b0 0x7f6d381446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d35d9b640 1 -- 172.21.15.7:0/3707899534 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d38141710 msgr2=0x7f6d3813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d35d9b640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d38141710 0x7f6d3813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d35d9b640 1 -- 172.21.15.7:0/3707899534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d3813c720 con 0x7f6d380b9ed0 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d35d9b640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 0x7f6d3813b250 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000c7c0 tx=0x7f6d2000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d2effd640 1 -- 172.21.15.7:0/3707899534 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d20007a30 con 0x7f6d380b9ed0 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d2effd640 1 -- 172.21.15.7:0/3707899534 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d20007bd0 con 0x7f6d380b9ed0 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.219+0000 7f6d2effd640 1 -- 172.21.15.7:0/3707899534 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d200113f0 con 0x7f6d380b9ed0 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.223+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d3813dbe0 con 0x7f6d380b9ed0 2024-03-19T01:23:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.223+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d3813a1b0 con 0x7f6d380b9ed0 2024-03-19T01:23:57.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.223+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d38075460 con 0x7f6d380b9ed0 2024-03-19T01:23:57.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.223+0000 7f6d2effd640 1 -- 172.21.15.7:0/3707899534 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6d20011590 con 0x7f6d380b9ed0 2024-03-19T01:23:57.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.223+0000 7f6d2effd640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6d18067660 0x7f6d18069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.223+0000 7f6d2effd640 1 -- 172.21.15.7:0/3707899534 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6d20085ab0 con 0x7f6d380b9ed0 2024-03-19T01:23:57.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.223+0000 7f6d3659c640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6d18067660 0x7f6d18069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.223+0000 7f6d3659c640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6d18067660 0x7f6d18069b20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d280ae000 tx=0x7f6d280a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:57.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.227+0000 7f6d2effd640 1 -- 172.21.15.7:0/3707899534 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6d20015020 con 0x7f6d380b9ed0 2024-03-19T01:23:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.379+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d380bc290 con 0x7f6d380b9ed0 2024-03-19T01:23:57.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d2effd640 1 -- 172.21.15.7:0/3707899534 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7f6d20056720 con 0x7f6d380b9ed0 2024-03-19T01:23:57.384 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:57.384 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6d18067660 msgr2=0x7f6d18069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6d18067660 0x7f6d18069b20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d280ae000 tx=0x7f6d280a7000 comp rx=0 tx=0).stop 2024-03-19T01:23:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 msgr2=0x7f6d3813b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 0x7f6d3813b250 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000c7c0 tx=0x7f6d2000cc80 comp rx=0 tx=0).stop 2024-03-19T01:23:57.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 shutdown_connections 2024-03-19T01:23:57.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6d18067660 0x7f6d18069b20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d38141710 0x7f6d3813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d380b9ed0 0x7f6d3813b250 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 --2- 172.21.15.7:0/3707899534 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d380b47b0 0x7f6d381446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 >> 172.21.15.7:0/3707899534 conn(0x7f6d38072df0 msgr2=0x7f6d381473a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:57.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 shutdown_connections 2024-03-19T01:23:57.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.383+0000 7f6d3c817640 1 -- 172.21.15.7:0/3707899534 wait complete. 2024-03-19T01:23:57.402 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:23:24.867856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:23:25.855754+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:23:57.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:23:57.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.511+0000 7f1a4b475640 1 Processor -- start 2024-03-19T01:23:57.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.511+0000 7f1a4b475640 1 -- start start 2024-03-19T01:23:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a4b475640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a440b5780 0x7f1a4414aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a4b475640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1a44149e50 0x7f1a4414a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a4b475640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f1a4406b2c0 0x7f1a4406b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a4b475640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f1a440b6250 con 0x7f1a44149e50 2024-03-19T01:23:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a4b475640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f1a440afa50 con 0x7f1a4406b2c0 2024-03-19T01:23:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a4b475640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a440afee0 con 0x7f1a440b5780 2024-03-19T01:23:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a48a10640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1a44149e50 0x7f1a4414a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a48a10640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1a44149e50 0x7f1a4414a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49654/0 (socket says 172.21.15.7:49654) 2024-03-19T01:23:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a48a10640 1 -- 172.21.15.7:0/2622185778 learned_addr learned my addr 172.21.15.7:0/2622185778 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a49211640 1 --2- 172.21.15.7:0/2622185778 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a440b5780 0x7f1a4414aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a49a12640 1 --2- 172.21.15.7:0/2622185778 >> v2:172.21.15.167:3300/0 conn(0x7f1a4406b2c0 0x7f1a4406b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a48a10640 1 -- 172.21.15.7:0/2622185778 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a440b5780 msgr2=0x7f1a4414aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a48a10640 1 --2- 172.21.15.7:0/2622185778 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a440b5780 0x7f1a4414aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a48a10640 1 -- 172.21.15.7:0/2622185778 >> v2:172.21.15.167:3300/0 conn(0x7f1a4406b2c0 msgr2=0x7f1a4406b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a48a10640 1 --2- 172.21.15.7:0/2622185778 >> v2:172.21.15.167:3300/0 conn(0x7f1a4406b2c0 0x7f1a4406b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a48a10640 1 -- 172.21.15.7:0/2622185778 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4414a7f0 con 0x7f1a44149e50 2024-03-19T01:23:57.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a48a10640 1 --2- 172.21.15.7:0/2622185778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 0x7f1a4414a2b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1a440702d0 tx=0x7f1a3401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea91d8366fe3a35 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:57.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a3bfff640 1 -- 172.21.15.7:0/2622185778 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a340b0070 con 0x7f1a44149e50 2024-03-19T01:23:57.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a3bfff640 1 -- 172.21.15.7:0/2622185778 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a340ab600 con 0x7f1a44149e50 2024-03-19T01:23:57.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a3bfff640 1 -- 172.21.15.7:0/2622185778 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a340ab9d0 con 0x7f1a44149e50 2024-03-19T01:23:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a4b475640 1 -- 172.21.15.7:0/2622185778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 msgr2=0x7f1a4414a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.515+0000 7f1a4b475640 1 --2- 172.21.15.7:0/2622185778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 0x7f1a4414a2b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1a440702d0 tx=0x7f1a3401c490 comp rx=0 tx=0).stop 2024-03-19T01:23:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- 172.21.15.7:0/2622185778 shutdown_connections 2024-03-19T01:23:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 --2- 172.21.15.7:0/2622185778 >> v2:172.21.15.167:3300/0 conn(0x7f1a4406b2c0 0x7f1a4406b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 --2- 172.21.15.7:0/2622185778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 0x7f1a4414a2b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 --2- 172.21.15.7:0/2622185778 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a440b5780 0x7f1a4414aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- 172.21.15.7:0/2622185778 >> 172.21.15.7:0/2622185778 conn(0x7f1a440b6610 msgr2=0x7f1a440b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- 172.21.15.7:0/2622185778 shutdown_connections 2024-03-19T01:23:57.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- 172.21.15.7:0/2622185778 wait complete. 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 Processor -- start 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- start start 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a4406b2c0 0x7f1a44070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a440b5780 0x7f1a441dffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 0x7f1a441e0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4406c040 con 0x7f1a44149e50 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4406d2a0 con 0x7f1a440b5780 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4406ec60 con 0x7f1a4406b2c0 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49a12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 0x7f1a441e0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49a12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 0x7f1a441e0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49668/0 (socket says 172.21.15.7:49668) 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49a12640 1 -- 172.21.15.7:0/707091915 learned_addr learned my addr 172.21.15.7:0/707091915 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49211640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a4406b2c0 0x7f1a44070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a48a10640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a440b5780 0x7f1a441dffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49a12640 1 -- 172.21.15.7:0/707091915 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a4406b2c0 msgr2=0x7f1a44070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49a12640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a4406b2c0 0x7f1a44070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49a12640 1 -- 172.21.15.7:0/707091915 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a440b5780 msgr2=0x7f1a441dffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49a12640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a440b5780 0x7f1a441dffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49a12640 1 -- 172.21.15.7:0/707091915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a44145a20 con 0x7f1a44149e50 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a49a12640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 0x7f1a441e0500 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c007e40 tx=0x7f1a3c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a39ffb640 1 -- 172.21.15.7:0/707091915 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c014620 con 0x7f1a44149e50 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a39ffb640 1 -- 172.21.15.7:0/707091915 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a3c019070 con 0x7f1a44149e50 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a39ffb640 1 -- 172.21.15.7:0/707091915 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c014a20 con 0x7f1a44149e50 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a44071d60 con 0x7f1a44149e50 2024-03-19T01:23:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a44071a70 con 0x7f1a44149e50 2024-03-19T01:23:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.519+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a44075570 con 0x7f1a44149e50 2024-03-19T01:23:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.523+0000 7f1a39ffb640 1 -- 172.21.15.7:0/707091915 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1a3c014bc0 con 0x7f1a44149e50 2024-03-19T01:23:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.523+0000 7f1a39ffb640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a28067660 0x7f1a28069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.523+0000 7f1a49211640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a28067660 0x7f1a28069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.523+0000 7f1a39ffb640 1 -- 172.21.15.7:0/707091915 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1a3c086060 con 0x7f1a44149e50 2024-03-19T01:23:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.523+0000 7f1a49211640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a28067660 0x7f1a28069b20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1a40006aa0 tx=0x7f1a40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:57.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.527+0000 7f1a39ffb640 1 -- 172.21.15.7:0/707091915 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1a3c056d70 con 0x7f1a44149e50 2024-03-19T01:23:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.775+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1a441dfd20 con 0x7f1a44149e50 2024-03-19T01:23:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.775+0000 7f1a39ffb640 1 -- 172.21.15.7:0/707091915 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f1a3c056670 con 0x7f1a44149e50 2024-03-19T01:23:57.778 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a28067660 msgr2=0x7f1a28069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a28067660 0x7f1a28069b20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1a40006aa0 tx=0x7f1a40008040 comp rx=0 tx=0).stop 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 msgr2=0x7f1a441e0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 0x7f1a441e0500 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c007e40 tx=0x7f1a3c00e4b0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 shutdown_connections 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a28067660 0x7f1a28069b20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a44149e50 0x7f1a441e0500 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a440b5780 0x7f1a441dffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 --2- 172.21.15.7:0/707091915 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a4406b2c0 0x7f1a44070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 >> 172.21.15.7:0/707091915 conn(0x7f1a440b6610 msgr2=0x7f1a44149340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 shutdown_connections 2024-03-19T01:23:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.779+0000 7f1a4b475640 1 -- 172.21.15.7:0/707091915 wait complete. 2024-03-19T01:23:57.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:23:57.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb13637a640 1 Processor -- start 2024-03-19T01:23:57.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb13637a640 1 -- start start 2024-03-19T01:23:57.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb13637a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 0x7fb1300b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb13637a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb13014a960 0x7fb13014ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb13637a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1300bd7e0 con 0x7fb130140f40 2024-03-19T01:23:57.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb13637a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fb1300b4f20 con 0x7fb13014a960 2024-03-19T01:23:57.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb13637a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb1300b6d70 con 0x7fb1300bd400 2024-03-19T01:23:57.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb127fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb13014a960 0x7fb13014ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb127fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb13014a960 0x7fb13014ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49682/0 (socket says 172.21.15.7:49682) 2024-03-19T01:23:57.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb127fff640 1 -- 172.21.15.7:0/745793485 learned_addr learned my addr 172.21.15.7:0/745793485 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:57.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb12ffff640 1 --2- 172.21.15.7:0/745793485 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 0x7fb1300b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb127fff640 1 -- 172.21.15.7:0/745793485 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 msgr2=0x7fb1300b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:57.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb127fff640 1 --2- 172.21.15.7:0/745793485 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 0x7fb1300b57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb127fff640 1 -- 172.21.15.7:0/745793485 >> v1:172.21.15.167:6789/0 conn(0x7fb130140f40 legacy=0x7fb1300b7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:57.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb127fff640 1 -- 172.21.15.7:0/745793485 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1300b5ce0 con 0x7fb13014a960 2024-03-19T01:23:57.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.915+0000 7fb127fff640 1 --2- 172.21.15.7:0/745793485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 0x7fb13014ade0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb130073910 tx=0x7fb11801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3dcd01dfc595c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:57.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb12f7fe640 1 -- 172.21.15.7:0/745793485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1180b0070 con 0x7fb13014a960 2024-03-19T01:23:57.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb12f7fe640 1 -- 172.21.15.7:0/745793485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1180ab600 con 0x7fb13014a960 2024-03-19T01:23:57.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb12f7fe640 1 -- 172.21.15.7:0/745793485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1180ab9d0 con 0x7fb13014a960 2024-03-19T01:23:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 -- 172.21.15.7:0/745793485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 msgr2=0x7fb13014ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 --2- 172.21.15.7:0/745793485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 0x7fb13014ade0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb130073910 tx=0x7fb11801c450 comp rx=0 tx=0).stop 2024-03-19T01:23:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 -- 172.21.15.7:0/745793485 shutdown_connections 2024-03-19T01:23:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 --2- 172.21.15.7:0/745793485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 0x7fb13014ade0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 --2- 172.21.15.7:0/745793485 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 0x7fb1300b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 -- 172.21.15.7:0/745793485 >> 172.21.15.7:0/745793485 conn(0x7fb130072df0 msgr2=0x7fb130073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 -- 172.21.15.7:0/745793485 shutdown_connections 2024-03-19T01:23:57.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 -- 172.21.15.7:0/745793485 wait complete. 2024-03-19T01:23:57.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 Processor -- start 2024-03-19T01:23:57.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 -- start start 2024-03-19T01:23:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 0x7fb1301e3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb130140f40 0x7fb130139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 0x7fb13013a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb1300b9440 con 0x7fb13014a960 2024-03-19T01:23:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb1300b92c0 con 0x7fb130140f40 2024-03-19T01:23:57.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.919+0000 7fb13637a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb130154d10 con 0x7fb1300bd400 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb134917640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 0x7fb13013a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb134917640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 0x7fb13013a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49686/0 (socket says 172.21.15.7:49686) 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb134917640 1 -- 172.21.15.7:0/2700601812 learned_addr learned my addr 172.21.15.7:0/2700601812 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12ffff640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 0x7fb1301e3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb127fff640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb130140f40 0x7fb130139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb134917640 1 -- 172.21.15.7:0/2700601812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 msgr2=0x7fb1301e3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb134917640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 0x7fb1301e3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb134917640 1 -- 172.21.15.7:0/2700601812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb130140f40 msgr2=0x7fb130139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb134917640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb130140f40 0x7fb130139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb134917640 1 -- 172.21.15.7:0/2700601812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13013c5f0 con 0x7fb13014a960 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12ffff640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 0x7fb1301e3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb134917640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 0x7fb13013a960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb12800c780 tx=0x7fb12800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12d7fa640 1 -- 172.21.15.7:0/2700601812 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128004b70 con 0x7fb13014a960 2024-03-19T01:23:57.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12d7fa640 1 -- 172.21.15.7:0/2700601812 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb128004d10 con 0x7fb13014a960 2024-03-19T01:23:57.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13013cdf0 con 0x7fb13014a960 2024-03-19T01:23:57.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12d7fa640 1 -- 172.21.15.7:0/2700601812 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12800aa90 con 0x7fb13014a960 2024-03-19T01:23:57.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb130147090 con 0x7fb13014a960 2024-03-19T01:23:57.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb13013a540 con 0x7fb13014a960 2024-03-19T01:23:57.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12d7fa640 1 -- 172.21.15.7:0/2700601812 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb12800ac30 con 0x7fb13014a960 2024-03-19T01:23:57.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12d7fa640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb110067690 0x7fb110069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:57.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12d7fa640 1 -- 172.21.15.7:0/2700601812 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb128085ba0 con 0x7fb13014a960 2024-03-19T01:23:57.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12ffff640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb110067690 0x7fb110069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:57.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.923+0000 7fb12ffff640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb110067690 0x7fb110069b50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb12000a9d0 tx=0x7fb120008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:57.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:57.927+0000 7fb12d7fa640 1 -- 172.21.15.7:0/2700601812 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb128020070 con 0x7fb13014a960 2024-03-19T01:23:58.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb130075570 con 0x7fb13014a960 2024-03-19T01:23:58.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb12d7fa640 1 -- 172.21.15.7:0/2700601812 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fb1280567a0 con 0x7fb13014a960 2024-03-19T01:23:58.120 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:23:58.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb110067690 msgr2=0x7fb110069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb110067690 0x7fb110069b50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb12000a9d0 tx=0x7fb120008040 comp rx=0 tx=0).stop 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 msgr2=0x7fb13013a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 0x7fb13013a960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb12800c780 tx=0x7fb12800cc40 comp rx=0 tx=0).stop 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 shutdown_connections 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb110067690 0x7fb110069b50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13014a960 0x7fb13013a960 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb130140f40 0x7fb130139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 --2- 172.21.15.7:0/2700601812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1300bd400 0x7fb1301e3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 >> 172.21.15.7:0/2700601812 conn(0x7fb130072df0 msgr2=0x7fb13014ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 shutdown_connections 2024-03-19T01:23:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.119+0000 7fb13637a640 1 -- 172.21.15.7:0/2700601812 wait complete. 2024-03-19T01:23:58.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:23:58.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f03130d9640 1 Processor -- start 2024-03-19T01:23:58.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f03130d9640 1 -- start start 2024-03-19T01:23:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f03130d9640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f030c0bb8f0 0x7f030c06da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f03130d9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f030c1413d0 0x7f030c141860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f03130d9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f030c06cc30 con 0x7f030c06d620 2024-03-19T01:23:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f03130d9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f030c0b5ac0 con 0x7f030c1413d0 2024-03-19T01:23:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f03130d9640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f030c0b6c90 con 0x7f030c0bb8f0 2024-03-19T01:23:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f0310e75640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f030c06d620 0x7f030c0b5ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35214/0 (socket says 172.21.15.7:35214) 2024-03-19T01:23:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f0310e75640 1 -- 172.21.15.7:0/3469961598 learned_addr learned my addr 172.21.15.7:0/3469961598 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f0311676640 1 --2- 172.21.15.7:0/3469961598 >> v2:172.21.15.7:3300/0 conn(0x7f030c1413d0 0x7f030c141860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f030bfff640 1 --2- 172.21.15.7:0/3469961598 >> v2:172.21.15.167:3300/0 conn(0x7f030c0bb8f0 0x7f030c06da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f0311676640 1 -- 172.21.15.7:0/3469961598 >> v2:172.21.15.167:3300/0 conn(0x7f030c0bb8f0 msgr2=0x7f030c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f0311676640 1 --2- 172.21.15.7:0/3469961598 >> v2:172.21.15.167:3300/0 conn(0x7f030c0bb8f0 0x7f030c06da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f0311676640 1 -- 172.21.15.7:0/3469961598 >> v1:172.21.15.7:6789/0 conn(0x7f030c06d620 legacy=0x7f030c0b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f0311676640 1 -- 172.21.15.7:0/3469961598 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030c0b65d0 con 0x7f030c1413d0 2024-03-19T01:23:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.251+0000 7f030b7fe640 1 -- 172.21.15.7:0/3469961598 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1231415235 0 0) 0x7f030c06cc30 con 0x7f030c06d620 2024-03-19T01:23:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f0311676640 1 --2- 172.21.15.7:0/3469961598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c1413d0 0x7f030c141860 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f030c072fb0 tx=0x7f030401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182a5a45a87044ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030b7fe640 1 -- 172.21.15.7:0/3469961598 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03040b0070 con 0x7f030c1413d0 2024-03-19T01:23:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030b7fe640 1 -- 172.21.15.7:0/3469961598 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03040ab600 con 0x7f030c1413d0 2024-03-19T01:23:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030b7fe640 1 -- 172.21.15.7:0/3469961598 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03040ab9d0 con 0x7f030c1413d0 2024-03-19T01:23:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 -- 172.21.15.7:0/3469961598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c1413d0 msgr2=0x7f030c141860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 --2- 172.21.15.7:0/3469961598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c1413d0 0x7f030c141860 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f030c072fb0 tx=0x7f030401c360 comp rx=0 tx=0).stop 2024-03-19T01:23:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 -- 172.21.15.7:0/3469961598 shutdown_connections 2024-03-19T01:23:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 --2- 172.21.15.7:0/3469961598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c1413d0 0x7f030c141860 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 --2- 172.21.15.7:0/3469961598 >> v2:172.21.15.167:3300/0 conn(0x7f030c0bb8f0 0x7f030c06da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 -- 172.21.15.7:0/3469961598 >> 172.21.15.7:0/3469961598 conn(0x7f030c0bace0 msgr2=0x7f030c0bb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 -- 172.21.15.7:0/3469961598 shutdown_connections 2024-03-19T01:23:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 -- 172.21.15.7:0/3469961598 wait complete. 2024-03-19T01:23:58.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 Processor -- start 2024-03-19T01:23:58.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 -- start start 2024-03-19T01:23:58.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c06d620 0x7f030c143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c0bb8f0 0x7f030c145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c1413d0 0x7f030c145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f030c154960 con 0x7f030c0bb8f0 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f030c1547e0 con 0x7f030c06d620 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f03130d9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f030c154ae0 con 0x7f030c1413d0 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c0bb8f0 0x7f030c145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c0bb8f0 0x7f030c145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49708/0 (socket says 172.21.15.7:49708) 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030bfff640 1 -- 172.21.15.7:0/1990937541 learned_addr learned my addr 172.21.15.7:0/1990937541 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f0311676640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c1413d0 0x7f030c145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030bfff640 1 -- 172.21.15.7:0/1990937541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c1413d0 msgr2=0x7f030c145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030bfff640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c1413d0 0x7f030c145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030bfff640 1 -- 172.21.15.7:0/1990937541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c06d620 msgr2=0x7f030c143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f0310e75640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c06d620 0x7f030c143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030bfff640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c06d620 0x7f030c143bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.255+0000 7f030bfff640 1 -- 172.21.15.7:0/1990937541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030c13aa40 con 0x7f030c0bb8f0 2024-03-19T01:23:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f030bfff640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c0bb8f0 0x7f030c145640 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f02f400d830 tx=0x7f02f400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f03097fa640 1 -- 172.21.15.7:0/1990937541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f400b660 con 0x7f030c0bb8f0 2024-03-19T01:23:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f03097fa640 1 -- 172.21.15.7:0/1990937541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02f4019070 con 0x7f030c0bb8f0 2024-03-19T01:23:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f030c13d5d0 con 0x7f030c0bb8f0 2024-03-19T01:23:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f03097fa640 1 -- 172.21.15.7:0/1990937541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f400ba00 con 0x7f030c0bb8f0 2024-03-19T01:23:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f030c13d2f0 con 0x7f030c0bb8f0 2024-03-19T01:23:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f030c13ad10 con 0x7f030c0bb8f0 2024-03-19T01:23:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f03097fa640 1 -- 172.21.15.7:0/1990937541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f02f401e460 con 0x7f030c0bb8f0 2024-03-19T01:23:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f03097fa640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f02ec067660 0x7f02ec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f0310e75640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f02ec067660 0x7f02ec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f03097fa640 1 -- 172.21.15.7:0/1990937541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f02f4085ec0 con 0x7f030c0bb8f0 2024-03-19T01:23:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.259+0000 7f0310e75640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f02ec067660 0x7f02ec069b20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0012f0 tx=0x7f02fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:58.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.267+0000 7f03097fa640 1 -- 172.21.15.7:0/1990937541 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f030c13ad10 con 0x7f030c0bb8f0 2024-03-19T01:23:58.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.551+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f030c075570 con 0x7f030c0bb8f0 2024-03-19T01:23:58.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.551+0000 7f03097fa640 1 -- 172.21.15.7:0/1990937541 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f02f4056bd0 con 0x7f030c0bb8f0 2024-03-19T01:23:58.554 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:23:58.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f02ec067660 msgr2=0x7f02ec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f02ec067660 0x7f02ec069b20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0012f0 tx=0x7f02fc009040 comp rx=0 tx=0).stop 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c0bb8f0 msgr2=0x7f030c145640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c0bb8f0 0x7f030c145640 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f02f400d830 tx=0x7f02f400dcf0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 shutdown_connections 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f02ec067660 0x7f02ec069b20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c1413d0 0x7f030c145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f030c0bb8f0 0x7f030c145640 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 --2- 172.21.15.7:0/1990937541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c06d620 0x7f030c143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 >> 172.21.15.7:0/1990937541 conn(0x7f030c0bace0 msgr2=0x7f030c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 shutdown_connections 2024-03-19T01:23:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.555+0000 7f03130d9640 1 -- 172.21.15.7:0/1990937541 wait complete. 2024-03-19T01:23:58.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f148780b640 1 Processor -- start 2024-03-19T01:23:58.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f148780b640 1 -- start start 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f148780b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f148006bee0 0x7f1480147c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f148780b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 0x7f1480151d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f148780b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14800aeef0 con 0x7f148009f060 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f148780b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f148006ba50 con 0x7f148006bee0 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f148780b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f148006de80 con 0x7f1480148170 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1484da6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f148006bee0 0x7f1480147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1484da6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f148006bee0 0x7f1480147c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49714/0 (socket says 172.21.15.7:49714) 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1484da6640 1 -- 172.21.15.7:0/1118793361 learned_addr learned my addr 172.21.15.7:0/1118793361 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1485da8640 1 --2- 172.21.15.7:0/1118793361 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 0x7f1480151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1484da6640 1 -- 172.21.15.7:0/1118793361 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 msgr2=0x7f1480151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1484da6640 1 --2- 172.21.15.7:0/1118793361 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 0x7f1480151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1484da6640 1 -- 172.21.15.7:0/1118793361 >> v1:172.21.15.167:6789/0 conn(0x7f148009f060 legacy=0x7f14800af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:58.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1484da6640 1 -- 172.21.15.7:0/1118793361 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f148009f440 con 0x7f148006bee0 2024-03-19T01:23:58.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1484da6640 1 --2- 172.21.15.7:0/1118793361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148006bee0 0x7f1480147c30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1470009870 tx=0x7f147001c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31bee0f10e848d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:58.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1477fff640 1 -- 172.21.15.7:0/1118793361 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14700b0070 con 0x7f148006bee0 2024-03-19T01:23:58.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1477fff640 1 -- 172.21.15.7:0/1118793361 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14700ab600 con 0x7f148006bee0 2024-03-19T01:23:58.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.683+0000 7f1477fff640 1 -- 172.21.15.7:0/1118793361 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14700ab9d0 con 0x7f148006bee0 2024-03-19T01:23:58.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- 172.21.15.7:0/1118793361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148006bee0 msgr2=0x7f1480147c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 --2- 172.21.15.7:0/1118793361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148006bee0 0x7f1480147c30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1470009870 tx=0x7f147001c2b0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- 172.21.15.7:0/1118793361 shutdown_connections 2024-03-19T01:23:58.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 --2- 172.21.15.7:0/1118793361 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 0x7f1480151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 --2- 172.21.15.7:0/1118793361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148006bee0 0x7f1480147c30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- 172.21.15.7:0/1118793361 >> 172.21.15.7:0/1118793361 conn(0x7f14800b6510 msgr2=0x7f14800b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:58.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- 172.21.15.7:0/1118793361 shutdown_connections 2024-03-19T01:23:58.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- 172.21.15.7:0/1118793361 wait complete. 2024-03-19T01:23:58.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 Processor -- start 2024-03-19T01:23:58.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- start start 2024-03-19T01:23:58.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f148006bee0 0x7f1480146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148009f060 0x7f14801dfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 0x7f14801e0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f148006ec00 con 0x7f148009f060 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1480148600 con 0x7f148006bee0 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f148006ed80 con 0x7f1480148170 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1484da6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148009f060 0x7f14801dfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1484da6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148009f060 0x7f14801dfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49722/0 (socket says 172.21.15.7:49722) 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1484da6640 1 -- 172.21.15.7:0/1066995456 learned_addr learned my addr 172.21.15.7:0/1066995456 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1485da8640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 0x7f14801e0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1484da6640 1 -- 172.21.15.7:0/1066995456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 msgr2=0x7f14801e0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1484da6640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 0x7f14801e0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1484da6640 1 -- 172.21.15.7:0/1066995456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f148006bee0 msgr2=0x7f1480146b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f14855a7640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f148006bee0 0x7f1480146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1484da6640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f148006bee0 0x7f1480146b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1484da6640 1 -- 172.21.15.7:0/1066995456 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14800b2da0 con 0x7f148009f060 2024-03-19T01:23:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f14855a7640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f148006bee0 0x7f1480146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1484da6640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148009f060 0x7f14801dfd30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f14700ae040 tx=0x7f147001aaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1475ffb640 1 -- 172.21.15.7:0/1066995456 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14700ba020 con 0x7f148009f060 2024-03-19T01:23:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1475ffb640 1 -- 172.21.15.7:0/1066995456 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14700b6640 con 0x7f148009f060 2024-03-19T01:23:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14800b1d70 con 0x7f148009f060 2024-03-19T01:23:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f1475ffb640 1 -- 172.21.15.7:0/1066995456 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14700b0040 con 0x7f148009f060 2024-03-19T01:23:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.687+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14800b3010 con 0x7f148009f060 2024-03-19T01:23:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.691+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14800b1b80 con 0x7f148009f060 2024-03-19T01:23:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.691+0000 7f1475ffb640 1 -- 172.21.15.7:0/1066995456 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f14700bd020 con 0x7f148009f060 2024-03-19T01:23:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.691+0000 7f1475ffb640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1464067690 0x7f1464069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.691+0000 7f14855a7640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1464067690 0x7f1464069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.691+0000 7f1475ffb640 1 -- 172.21.15.7:0/1066995456 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f147011fc90 con 0x7f148009f060 2024-03-19T01:23:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.691+0000 7f14855a7640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1464067690 0x7f1464069b50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f147c003e40 tx=0x7f147c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:58.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.695+0000 7f1475ffb640 1 -- 172.21.15.7:0/1066995456 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f14700c4030 con 0x7f148009f060 2024-03-19T01:23:58.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.891+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1480075460 con 0x7f148009f060 2024-03-19T01:23:58.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.891+0000 7f1475ffb640 1 -- 172.21.15.7:0/1066995456 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3524 (secure 0 0 0) 0x7f14800aeef0 con 0x7f148009f060 2024-03-19T01:23:58.893 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:58.894 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4208,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/656985991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":656985991},{"type":"v1","addr":"172.21.15.167:6833","nonce":656985991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:23:26.868603+0000","modified":"2024-03-19T01:23:29.918429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-03-19T01:23:58.894 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1464067690 msgr2=0x7f1464069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1464067690 0x7f1464069b50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f147c003e40 tx=0x7f147c009040 comp rx=0 tx=0).stop 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148009f060 msgr2=0x7f14801dfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148009f060 0x7f14801dfd30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f14700ae040 tx=0x7f147001aaf0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 shutdown_connections 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1464067690 0x7f1464069b50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1480148170 0x7f14801e0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f148009f060 0x7f14801dfd30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 --2- 172.21.15.7:0/1066995456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f148006bee0 0x7f1480146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 >> 172.21.15.7:0/1066995456 conn(0x7f14800b6510 msgr2=0x7f1480143510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:58.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 shutdown_connections 2024-03-19T01:23:58.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:58.895+0000 7f148780b640 1 -- 172.21.15.7:0/1066995456 wait complete. 2024-03-19T01:23:58.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:23:59.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de3eac640 1 Processor -- start 2024-03-19T01:23:59.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de3eac640 1 -- start start 2024-03-19T01:23:59.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de3eac640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6ddc0b3a20 0x7f6ddc150c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de3eac640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ddc072a90 con 0x7f6ddc0726b0 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de3eac640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ddc0b4a30 con 0x7f6ddc0ba710 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de3eac640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f6ddc0b4750 con 0x7f6ddc0b3a20 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de1c48640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6ddc0726b0 0x7f6ddc148940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35230/0 (socket says 172.21.15.7:35230) 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de1c48640 1 -- 172.21.15.7:0/2971305505 learned_addr learned my addr 172.21.15.7:0/2971305505 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de2449640 1 --2- 172.21.15.7:0/2971305505 >> v2:172.21.15.7:3300/0 conn(0x7f6ddc0b3a20 0x7f6ddc150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de0c46640 1 -- 172.21.15.7:0/2971305505 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583206753 0 0) 0x7f6ddc072a90 con 0x7f6ddc0726b0 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de0c46640 1 -- 172.21.15.7:0/2971305505 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dc8003530 con 0x7f6ddc0726b0 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de2449640 1 -- 172.21.15.7:0/2971305505 >> v1:172.21.15.167:6789/0 conn(0x7f6ddc0ba710 legacy=0x7f6ddc0b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de2449640 1 -- 172.21.15.7:0/2971305505 >> v1:172.21.15.7:6789/0 conn(0x7f6ddc0726b0 legacy=0x7f6ddc148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de2449640 1 -- 172.21.15.7:0/2971305505 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc0b5470 con 0x7f6ddc0b3a20 2024-03-19T01:23:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.035+0000 7f6de2449640 1 --2- 172.21.15.7:0/2971305505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0b3a20 0x7f6ddc150c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009d90 tx=0x7f6dd401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926c85a05bc223a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de0c46640 1 -- 172.21.15.7:0/2971305505 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40b2070 con 0x7f6ddc0b3a20 2024-03-19T01:23:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de0c46640 1 -- 172.21.15.7:0/2971305505 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd40a5930 con 0x7f6ddc0b3a20 2024-03-19T01:23:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de0c46640 1 -- 172.21.15.7:0/2971305505 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40a5d00 con 0x7f6ddc0b3a20 2024-03-19T01:23:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 -- 172.21.15.7:0/2971305505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0b3a20 msgr2=0x7f6ddc150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 --2- 172.21.15.7:0/2971305505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0b3a20 0x7f6ddc150c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009d90 tx=0x7f6dd401c8b0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 -- 172.21.15.7:0/2971305505 shutdown_connections 2024-03-19T01:23:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 --2- 172.21.15.7:0/2971305505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0b3a20 0x7f6ddc150c80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 -- 172.21.15.7:0/2971305505 >> 172.21.15.7:0/2971305505 conn(0x7f6ddc074130 msgr2=0x7f6ddc074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 -- 172.21.15.7:0/2971305505 shutdown_connections 2024-03-19T01:23:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 -- 172.21.15.7:0/2971305505 wait complete. 2024-03-19T01:23:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 Processor -- start 2024-03-19T01:23:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 -- start start 2024-03-19T01:23:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ddc0726b0 0x7f6ddc071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ddc0b3a20 0x7f6ddc14f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0ba710 0x7f6ddc14e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc153b60 con 0x7f6ddc0ba710 2024-03-19T01:23:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc1539e0 con 0x7f6ddc0726b0 2024-03-19T01:23:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.039+0000 7f6de3eac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6ddc153ce0 con 0x7f6ddc0b3a20 2024-03-19T01:23:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de2449640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0ba710 0x7f6ddc14e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de2449640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0ba710 0x7f6ddc14e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49738/0 (socket says 172.21.15.7:49738) 2024-03-19T01:23:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de2449640 1 -- 172.21.15.7:0/169110011 learned_addr learned my addr 172.21.15.7:0/169110011 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de1c48640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ddc0726b0 0x7f6ddc071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de1447640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ddc0b3a20 0x7f6ddc14f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de2449640 1 -- 172.21.15.7:0/169110011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ddc0b3a20 msgr2=0x7f6ddc14f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de2449640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ddc0b3a20 0x7f6ddc14f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de2449640 1 -- 172.21.15.7:0/169110011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ddc0726b0 msgr2=0x7f6ddc071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de2449640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ddc0726b0 0x7f6ddc071660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de2449640 1 -- 172.21.15.7:0/169110011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc14e9f0 con 0x7f6ddc0ba710 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de1447640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ddc0b3a20 0x7f6ddc14f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de2449640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0ba710 0x7f6ddc14e1c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6dd401a4e0 tx=0x7f6dd4019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6dce7fc640 1 -- 172.21.15.7:0/169110011 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40b2050 con 0x7f6ddc0ba710 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6dce7fc640 1 -- 172.21.15.7:0/169110011 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd40be070 con 0x7f6ddc0ba710 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ddc14ecd0 con 0x7f6ddc0ba710 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6dce7fc640 1 -- 172.21.15.7:0/169110011 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40ada50 con 0x7f6ddc0ba710 2024-03-19T01:23:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ddc14e700 con 0x7f6ddc0ba710 2024-03-19T01:23:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ddc075460 con 0x7f6ddc0ba710 2024-03-19T01:23:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6dce7fc640 1 -- 172.21.15.7:0/169110011 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6dd40adbf0 con 0x7f6ddc0ba710 2024-03-19T01:23:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6dce7fc640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6dc0067660 0x7f6dc0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de1c48640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6dc0067660 0x7f6dc0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6dce7fc640 1 -- 172.21.15.7:0/169110011 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6dd4121520 con 0x7f6ddc0ba710 2024-03-19T01:23:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.043+0000 7f6de1c48640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6dc0067660 0x7f6dc0069b20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80011f0 tx=0x7f6dd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.047+0000 7f6dce7fc640 1 -- 172.21.15.7:0/169110011 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6dd4016560 con 0x7f6ddc0ba710 2024-03-19T01:23:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.391+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ddc0b8300 con 0x7f6ddc0ba710 2024-03-19T01:23:59.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.395+0000 7f6dce7fc640 1 -- 172.21.15.7:0/169110011 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7f6dd40f2230 con 0x7f6ddc0ba710 2024-03-19T01:23:59.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:59.397 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:23:59.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6dc0067660 msgr2=0x7f6dc0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6dc0067660 0x7f6dc0069b20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80011f0 tx=0x7f6dd8009040 comp rx=0 tx=0).stop 2024-03-19T01:23:59.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0ba710 msgr2=0x7f6ddc14e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0ba710 0x7f6ddc14e1c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6dd401a4e0 tx=0x7f6dd4019650 comp rx=0 tx=0).stop 2024-03-19T01:23:59.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 shutdown_connections 2024-03-19T01:23:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6dc0067660 0x7f6dc0069b20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ddc0b3a20 0x7f6ddc14f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc0ba710 0x7f6ddc14e1c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 --2- 172.21.15.7:0/169110011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ddc0726b0 0x7f6ddc071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 >> 172.21.15.7:0/169110011 conn(0x7f6ddc074130 msgr2=0x7f6ddc067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 shutdown_connections 2024-03-19T01:23:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.399+0000 7f6de3eac640 1 -- 172.21.15.7:0/169110011 wait complete. 2024-03-19T01:23:59.424 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-19T01:23:59.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:59.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb1409a5640 1 Processor -- start 2024-03-19T01:23:59.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb1409a5640 1 -- start start 2024-03-19T01:23:59.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb1409a5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fb13c0b81c0 0x7fb13c1486e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb1409a5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb13c06f1c0 0x7fb13c13f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb1409a5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb13c0b85a0 con 0x7fb13c06f640 2024-03-19T01:23:59.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb1409a5640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fb13c0b2710 con 0x7fb13c06f1c0 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb1409a5640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fb13c070960 con 0x7fb13c0b81c0 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb139d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb13c06f1c0 0x7fb13c13f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb139d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb13c06f1c0 0x7fb13c13f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49750/0 (socket says 172.21.15.7:49750) 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb139d9b640 1 -- 172.21.15.7:0/983993988 learned_addr learned my addr 172.21.15.7:0/983993988 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb13a59c640 1 --2- 172.21.15.7:0/983993988 >> v2:172.21.15.167:3300/0 conn(0x7fb13c0b81c0 0x7fb13c1486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb139d9b640 1 -- 172.21.15.7:0/983993988 >> v2:172.21.15.167:3300/0 conn(0x7fb13c0b81c0 msgr2=0x7fb13c1486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb139d9b640 1 --2- 172.21.15.7:0/983993988 >> v2:172.21.15.167:3300/0 conn(0x7fb13c0b81c0 0x7fb13c1486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb139d9b640 1 -- 172.21.15.7:0/983993988 >> v1:172.21.15.167:6789/0 conn(0x7fb13c06f640 legacy=0x7fb13c145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb139d9b640 1 -- 172.21.15.7:0/983993988 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c0b20c0 con 0x7fb13c06f1c0 2024-03-19T01:23:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.559+0000 7fb13a59c640 1 --2- 172.21.15.7:0/983993988 >> v2:172.21.15.167:3300/0 conn(0x7fb13c0b81c0 0x7fb13c1486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb139d9b640 1 --2- 172.21.15.7:0/983993988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f1c0 0x7fb13c13f0a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0b9950 tx=0x7fb13001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130e7b0b6b26c906 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb13959a640 1 -- 172.21.15.7:0/983993988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1300ab640 con 0x7fb13c06f1c0 2024-03-19T01:23:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb13959a640 1 -- 172.21.15.7:0/983993988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1300b0070 con 0x7fb13c06f1c0 2024-03-19T01:23:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb13959a640 1 -- 172.21.15.7:0/983993988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1300aba00 con 0x7fb13c06f1c0 2024-03-19T01:23:59.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb1409a5640 1 -- 172.21.15.7:0/983993988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f1c0 msgr2=0x7fb13c13f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb1409a5640 1 --2- 172.21.15.7:0/983993988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f1c0 0x7fb13c13f0a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0b9950 tx=0x7fb13001c420 comp rx=0 tx=0).stop 2024-03-19T01:23:59.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb1409a5640 1 -- 172.21.15.7:0/983993988 shutdown_connections 2024-03-19T01:23:59.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb1409a5640 1 --2- 172.21.15.7:0/983993988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f1c0 0x7fb13c13f0a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb1409a5640 1 --2- 172.21.15.7:0/983993988 >> v2:172.21.15.167:3300/0 conn(0x7fb13c0b81c0 0x7fb13c1486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb1409a5640 1 -- 172.21.15.7:0/983993988 >> 172.21.15.7:0/983993988 conn(0x7fb13c0b5330 msgr2=0x7fb13c0b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb1409a5640 1 -- 172.21.15.7:0/983993988 shutdown_connections 2024-03-19T01:23:59.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.563+0000 7fb1409a5640 1 -- 172.21.15.7:0/983993988 wait complete. 2024-03-19T01:23:59.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 Processor -- start 2024-03-19T01:23:59.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 -- start start 2024-03-19T01:23:59.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13c06f1c0 0x7fb13c0b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f640 0x7fb13c1e0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb13c0b81c0 0x7fb13c13db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c152a90 con 0x7fb13c06f640 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c152c10 con 0x7fb13c06f1c0 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb13c152d90 con 0x7fb13c0b81c0 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb139d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f640 0x7fb13c1e0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb139d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f640 0x7fb13c1e0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49762/0 (socket says 172.21.15.7:49762) 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb139d9b640 1 -- 172.21.15.7:0/1939359870 learned_addr learned my addr 172.21.15.7:0/1939359870 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb13a59c640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13c06f1c0 0x7fb13c0b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb13ad9d640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb13c0b81c0 0x7fb13c13db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb139d9b640 1 -- 172.21.15.7:0/1939359870 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb13c0b81c0 msgr2=0x7fb13c13db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb139d9b640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb13c0b81c0 0x7fb13c13db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb139d9b640 1 -- 172.21.15.7:0/1939359870 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13c06f1c0 msgr2=0x7fb13c0b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb139d9b640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13c06f1c0 0x7fb13c0b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb139d9b640 1 -- 172.21.15.7:0/1939359870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c13c9d0 con 0x7fb13c06f640 2024-03-19T01:23:59.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb13a59c640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13c06f1c0 0x7fb13c0b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:59.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb139d9b640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f640 0x7fb13c1e0090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb13001ccc0 tx=0x7fb13001ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb12affd640 1 -- 172.21.15.7:0/1939359870 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1300be3f0 con 0x7fb13c06f640 2024-03-19T01:23:59.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb12affd640 1 -- 172.21.15.7:0/1939359870 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1300be590 con 0x7fb13c06f640 2024-03-19T01:23:59.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c13ccb0 con 0x7fb13c06f640 2024-03-19T01:23:59.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb12affd640 1 -- 172.21.15.7:0/1939359870 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1300be960 con 0x7fb13c06f640 2024-03-19T01:23:59.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c0aff80 con 0x7fb13c06f640 2024-03-19T01:23:59.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.567+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb13c075460 con 0x7fb13c06f640 2024-03-19T01:23:59.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.571+0000 7fb12affd640 1 -- 172.21.15.7:0/1939359870 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb1300c7430 con 0x7fb13c06f640 2024-03-19T01:23:59.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.571+0000 7fb12affd640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb11c067690 0x7fb11c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.571+0000 7fb12affd640 1 -- 172.21.15.7:0/1939359870 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb130120440 con 0x7fb13c06f640 2024-03-19T01:23:59.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.571+0000 7fb13a59c640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb11c067690 0x7fb11c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.571+0000 7fb13a59c640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb11c067690 0x7fb11c069b50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0079d0 tx=0x7fb12c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.575+0000 7fb12affd640 1 -- 172.21.15.7:0/1939359870 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb1300160d0 con 0x7fb13c06f640 2024-03-19T01:23:59.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.767+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb13c13acc0 con 0x7fb13c06f640 2024-03-19T01:23:59.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb12affd640 1 -- 172.21.15.7:0/1939359870 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3524 (secure 0 0 0) 0x7fb1300f1130 con 0x7fb13c06f640 2024-03-19T01:23:59.771 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:23:59.771 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4208,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/656985991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":656985991},{"type":"v1","addr":"172.21.15.167:6833","nonce":656985991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:23:26.868603+0000","modified":"2024-03-19T01:23:29.918429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-03-19T01:23:59.772 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-03-19T01:23:59.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb11c067690 msgr2=0x7fb11c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb11c067690 0x7fb11c069b50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0079d0 tx=0x7fb12c008040 comp rx=0 tx=0).stop 2024-03-19T01:23:59.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f640 msgr2=0x7fb13c1e0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f640 0x7fb13c1e0090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb13001ccc0 tx=0x7fb13001ac60 comp rx=0 tx=0).stop 2024-03-19T01:23:59.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 shutdown_connections 2024-03-19T01:23:59.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb11c067690 0x7fb11c069b50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb13c0b81c0 0x7fb13c13db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c06f640 0x7fb13c1e0090 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 --2- 172.21.15.7:0/1939359870 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13c06f1c0 0x7fb13c0b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 >> 172.21.15.7:0/1939359870 conn(0x7fb13c0b5330 msgr2=0x7fb13c0b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 shutdown_connections 2024-03-19T01:23:59.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.771+0000 7fb1409a5640 1 -- 172.21.15.7:0/1939359870 wait complete. 2024-03-19T01:23:59.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:23:59.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9f33e640 1 Processor -- start 2024-03-19T01:23:59.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9f33e640 1 -- start start 2024-03-19T01:23:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9f33e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d980ba710 0x7f1d980b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9f33e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d98072a90 con 0x7f1d980726b0 2024-03-19T01:23:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9f33e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d980b4a30 con 0x7f1d980b5c70 2024-03-19T01:23:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9f33e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1d980b4750 con 0x7f1d980ba710 2024-03-19T01:23:59.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9d0da640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1d980726b0 0x7f1d98148940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35242/0 (socket says 172.21.15.7:35242) 2024-03-19T01:23:59.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9d0da640 1 -- 172.21.15.7:0/2199160960 learned_addr learned my addr 172.21.15.7:0/2199160960 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:23:59.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9c8d9640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d980ba710 0x7f1d980b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992402467 0 0) 0x7f1d98072a90 con 0x7f1d980726b0 2024-03-19T01:23:59.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d80003530 con 0x7f1d980726b0 2024-03-19T01:23:59.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1857845900 0 0) 0x7f1d80003530 con 0x7f1d980726b0 2024-03-19T01:23:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d980ba710 msgr2=0x7f1d980b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d980ba710 0x7f1d980b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 >> v1:172.21.15.167:6789/0 conn(0x7f1d980b5c70 legacy=0x7f1d9814fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d980b5470 con 0x7f1d980726b0 2024-03-19T01:23:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f1d88001a00 con 0x7f1d980726b0 2024-03-19T01:23:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 >> v1:172.21.15.7:6789/0 conn(0x7f1d980726b0 legacy=0x7f1d98148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d800039f0 0x7f1d80003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d80004810 0x7f1d80004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d980ba710 0x7f1d80005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1d980b4750 con 0x7f1d800039f0 2024-03-19T01:23:59.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8000bb00 con 0x7f1d80004810 2024-03-19T01:23:59.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8000bd00 con 0x7f1d980ba710 2024-03-19T01:23:59.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9d0da640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d800039f0 0x7f1d80003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9d8db640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d980ba710 0x7f1d80005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9c8d9640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d80004810 0x7f1d80004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9d0da640 1 -- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d980ba710 msgr2=0x7f1d80005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9d0da640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d980ba710 0x7f1d80005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9d0da640 1 -- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d80004810 msgr2=0x7f1d80004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9d0da640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d80004810 0x7f1d80004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9d0da640 1 -- 172.21.15.7:0/2199160960 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d80003530 con 0x7f1d800039f0 2024-03-19T01:23:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.919+0000 7f1d9c8d9640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d80004810 0x7f1d80004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:23:59.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9d0da640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d800039f0 0x7f1d80003e10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800a860 tx=0x7f1d8801d350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e6fd76de7990e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8800a480 con 0x7f1d800039f0 2024-03-19T01:23:59.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d880ad630 con 0x7f1d800039f0 2024-03-19T01:23:59.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d97fff640 1 -- 172.21.15.7:0/2199160960 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d880ada60 con 0x7f1d800039f0 2024-03-19T01:23:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 -- 172.21.15.7:0/2199160960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d800039f0 msgr2=0x7f1d80003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 --2- 172.21.15.7:0/2199160960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d800039f0 0x7f1d80003e10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800a860 tx=0x7f1d8801d350 comp rx=0 tx=0).stop 2024-03-19T01:23:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9d8db640 1 -- 172.21.15.7:0/2199160960 reap_dead start 2024-03-19T01:23:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 -- 172.21.15.7:0/2199160960 shutdown_connections 2024-03-19T01:23:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 -- 172.21.15.7:0/2199160960 >> 172.21.15.7:0/2199160960 conn(0x7f1d98074130 msgr2=0x7f1d98074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 -- 172.21.15.7:0/2199160960 shutdown_connections 2024-03-19T01:23:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 -- 172.21.15.7:0/2199160960 wait complete. 2024-03-19T01:23:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 Processor -- start 2024-03-19T01:23:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 -- start start 2024-03-19T01:23:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d800039f0 0x7f1d98145b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d9814e200 0x7f1d981ddad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d9813dda0 0x7f1d9813e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1d98151510 con 0x7f1d9813dda0 2024-03-19T01:23:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1d98151390 con 0x7f1d800039f0 2024-03-19T01:23:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9f33e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1d98151690 con 0x7f1d9814e200 2024-03-19T01:23:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9d8db640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d9813dda0 0x7f1d9813e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9d8db640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d9813dda0 0x7f1d9813e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49772/0 (socket says 172.21.15.7:49772) 2024-03-19T01:23:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9d8db640 1 -- 172.21.15.7:0/870716037 learned_addr learned my addr 172.21.15.7:0/870716037 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:23:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9d0da640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d800039f0 0x7f1d98145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.923+0000 7f1d9c8d9640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d9814e200 0x7f1d981ddad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9d8db640 1 -- 172.21.15.7:0/870716037 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d9814e200 msgr2=0x7f1d981ddad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9d8db640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d9814e200 0x7f1d981ddad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9d8db640 1 -- 172.21.15.7:0/870716037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d800039f0 msgr2=0x7f1d98145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9d8db640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d800039f0 0x7f1d98145b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9d8db640 1 -- 172.21.15.7:0/870716037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d98142530 con 0x7f1d9813dda0 2024-03-19T01:23:59.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9d8db640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d9813dda0 0x7f1d9813e220 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1d90000c10 tx=0x7f1d9000e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d95ffb640 1 -- 172.21.15.7:0/870716037 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d90019070 con 0x7f1d9813dda0 2024-03-19T01:23:59.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d95ffb640 1 -- 172.21.15.7:0/870716037 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d90007e30 con 0x7f1d9813dda0 2024-03-19T01:23:59.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d9814f390 con 0x7f1d9813dda0 2024-03-19T01:23:59.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d95ffb640 1 -- 172.21.15.7:0/870716037 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d90003600 con 0x7f1d9813dda0 2024-03-19T01:23:59.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d98142250 con 0x7f1d9813dda0 2024-03-19T01:23:59.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d98075460 con 0x7f1d9813dda0 2024-03-19T01:23:59.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d95ffb640 1 -- 172.21.15.7:0/870716037 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1d9001c070 con 0x7f1d9813dda0 2024-03-19T01:23:59.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d95ffb640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1d84067690 0x7f1d84069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9d0da640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1d84067690 0x7f1d84069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d95ffb640 1 -- 172.21.15.7:0/870716037 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1d90084a90 con 0x7f1d9813dda0 2024-03-19T01:23:59.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.927+0000 7f1d9d0da640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1d84067690 0x7f1d84069b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1d880ad000 tx=0x7f1d880af040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:23:59.931+0000 7f1d95ffb640 1 -- 172.21.15.7:0/870716037 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1d900557a0 con 0x7f1d9813dda0 2024-03-19T01:24:00.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d9813fa80 con 0x7f1d9813dda0 2024-03-19T01:24:00.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d95ffb640 1 -- 172.21.15.7:0/870716037 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3524 (secure 0 0 0) 0x7f1d900550a0 con 0x7f1d9813dda0 2024-03-19T01:24:00.145 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:00.145 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4208,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/656985991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":656985991},{"type":"v1","addr":"172.21.15.167:6833","nonce":656985991}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:23:26.868603+0000","modified":"2024-03-19T01:23:29.918429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1481596897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1481596897},{"type":"v1","addr":"172.21.15.7:6835","nonce":1481596897}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4266,"qdb_cluster":[4266]},"id":1}]} 2024-03-19T01:24:00.145 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-03-19T01:24:00.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1d84067690 msgr2=0x7f1d84069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1d84067690 0x7f1d84069b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1d880ad000 tx=0x7f1d880af040 comp rx=0 tx=0).stop 2024-03-19T01:24:00.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d9813dda0 msgr2=0x7f1d9813e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d9813dda0 0x7f1d9813e220 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1d90000c10 tx=0x7f1d9000e8a0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 shutdown_connections 2024-03-19T01:24:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1d84067690 0x7f1d84069b50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d9814e200 0x7f1d981ddad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d9813dda0 0x7f1d9813e220 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 --2- 172.21.15.7:0/870716037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d800039f0 0x7f1d98145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 >> 172.21.15.7:0/870716037 conn(0x7f1d98074130 msgr2=0x7f1d980b9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:00.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 shutdown_connections 2024-03-19T01:24:00.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.143+0000 7f1d9f33e640 1 -- 172.21.15.7:0/870716037 wait complete. 2024-03-19T01:24:00.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:00.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.275+0000 7efdc34e3640 1 Processor -- start 2024-03-19T01:24:00.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.275+0000 7efdc34e3640 1 -- start start 2024-03-19T01:24:00.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efdbc072aa0 0x7efdbc14a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdbc0732d0 con 0x7efdbc06df40 2024-03-19T01:24:00.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdbc06ca50 con 0x7efdbc072e80 2024-03-19T01:24:00.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7efdbc06e3b0 con 0x7efdbc072aa0 2024-03-19T01:24:00.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc127f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efdbc072aa0 0x7efdbc14a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc0a7e640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efdbc06df40 0x7efdbc0b6a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35244/0 (socket says 172.21.15.7:35244) 2024-03-19T01:24:00.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc0a7e640 1 -- 172.21.15.7:0/2208538231 learned_addr learned my addr 172.21.15.7:0/2208538231 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:00.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc127f640 1 -- 172.21.15.7:0/2208538231 >> v1:172.21.15.167:6789/0 conn(0x7efdbc072e80 legacy=0x7efdbc151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:00.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc127f640 1 -- 172.21.15.7:0/2208538231 >> v1:172.21.15.7:6789/0 conn(0x7efdbc06df40 legacy=0x7efdbc0b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc127f640 1 -- 172.21.15.7:0/2208538231 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdbc06da20 con 0x7efdbc072aa0 2024-03-19T01:24:00.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc127f640 1 --2- 172.21.15.7:0/2208538231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072aa0 0x7efdbc14a9a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efdac0097c0 tx=0x7efdac01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9e78ecd3b0fbf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:00.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdb3fff640 1 -- 172.21.15.7:0/2208538231 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac0b0070 con 0x7efdbc072aa0 2024-03-19T01:24:00.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdb3fff640 1 -- 172.21.15.7:0/2208538231 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdac0ab600 con 0x7efdbc072aa0 2024-03-19T01:24:00.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdb3fff640 1 -- 172.21.15.7:0/2208538231 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac0ab9d0 con 0x7efdbc072aa0 2024-03-19T01:24:00.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 -- 172.21.15.7:0/2208538231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072aa0 msgr2=0x7efdbc14a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 --2- 172.21.15.7:0/2208538231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072aa0 0x7efdbc14a9a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efdac0097c0 tx=0x7efdac01c460 comp rx=0 tx=0).stop 2024-03-19T01:24:00.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 -- 172.21.15.7:0/2208538231 shutdown_connections 2024-03-19T01:24:00.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 --2- 172.21.15.7:0/2208538231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072aa0 0x7efdbc14a9a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 -- 172.21.15.7:0/2208538231 >> 172.21.15.7:0/2208538231 conn(0x7efdbc06bc10 msgr2=0x7efdbc0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:00.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 -- 172.21.15.7:0/2208538231 shutdown_connections 2024-03-19T01:24:00.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.279+0000 7efdc34e3640 1 -- 172.21.15.7:0/2208538231 wait complete. 2024-03-19T01:24:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 Processor -- start 2024-03-19T01:24:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 -- start start 2024-03-19T01:24:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdbc06df40 0x7efdbc139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdbc072aa0 0x7efdbc145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072e80 0x7efdbc0b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc155df0 con 0x7efdbc072e80 2024-03-19T01:24:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc155c70 con 0x7efdbc06df40 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efdbc155f70 con 0x7efdbc072aa0 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc1a80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072e80 0x7efdbc0b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc1a80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072e80 0x7efdbc0b27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49792/0 (socket says 172.21.15.7:49792) 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc1a80640 1 -- 172.21.15.7:0/2846632317 learned_addr learned my addr 172.21.15.7:0/2846632317 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc127f640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdbc06df40 0x7efdbc139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc0a7e640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdbc072aa0 0x7efdbc145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc1a80640 1 -- 172.21.15.7:0/2846632317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdbc072aa0 msgr2=0x7efdbc145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc1a80640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdbc072aa0 0x7efdbc145b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc1a80640 1 -- 172.21.15.7:0/2846632317 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdbc06df40 msgr2=0x7efdbc139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc1a80640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdbc06df40 0x7efdbc139df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc1a80640 1 -- 172.21.15.7:0/2846632317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdbc13aa10 con 0x7efdbc072e80 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc127f640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdbc06df40 0x7efdbc139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc1a80640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072e80 0x7efdbc0b27f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c780 tx=0x7efdb400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdb1ffb640 1 -- 172.21.15.7:0/2846632317 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4004b70 con 0x7efdbc072e80 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdb1ffb640 1 -- 172.21.15.7:0/2846632317 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdb4004d10 con 0x7efdbc072e80 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdbc0b35a0 con 0x7efdbc072e80 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdb1ffb640 1 -- 172.21.15.7:0/2846632317 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb400aa90 con 0x7efdbc072e80 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.283+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdbc0b3a30 con 0x7efdbc072e80 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.287+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdbc075460 con 0x7efdbc072e80 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.287+0000 7efdb1ffb640 1 -- 172.21.15.7:0/2846632317 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7efdb400ac30 con 0x7efdbc072e80 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.287+0000 7efdb1ffb640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efda0067660 0x7efda0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.287+0000 7efdb1ffb640 1 -- 172.21.15.7:0/2846632317 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efdb4085ae0 con 0x7efdbc072e80 2024-03-19T01:24:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.287+0000 7efdc127f640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efda0067660 0x7efda0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.287+0000 7efdc127f640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efda0067660 0x7efda0069b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efdac0ad000 tx=0x7efdac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:00.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.291+0000 7efdb1ffb640 1 -- 172.21.15.7:0/2846632317 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7efdb4020070 con 0x7efdbc072e80 2024-03-19T01:24:00.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.455+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efdbc06d370 con 0x7efdbc072e80 2024-03-19T01:24:00.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.455+0000 7efdb1ffb640 1 -- 172.21.15.7:0/2846632317 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16569 (secure 0 0 0) 0x7efdb4056750 con 0x7efdbc072e80 2024-03-19T01:24:00.458 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:00.458 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:23:28.490943+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:23:24.867856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:23:25.855754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efda0067660 msgr2=0x7efda0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efda0067660 0x7efda0069b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efdac0ad000 tx=0x7efdac008040 comp rx=0 tx=0).stop 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072e80 msgr2=0x7efdbc0b27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072e80 0x7efdbc0b27f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c780 tx=0x7efdb400cc40 comp rx=0 tx=0).stop 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 shutdown_connections 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7efda0067660 0x7efda0069b20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdbc072aa0 0x7efdbc145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efdbc072e80 0x7efdbc0b27f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 --2- 172.21.15.7:0/2846632317 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdbc06df40 0x7efdbc139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 >> 172.21.15.7:0/2846632317 conn(0x7efdbc06bc10 msgr2=0x7efdbc0b9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 shutdown_connections 2024-03-19T01:24:00.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.459+0000 7efdc34e3640 1 -- 172.21.15.7:0/2846632317 wait complete. 2024-03-19T01:24:00.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-19T01:24:00.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.587+0000 7fb68429e640 1 Processor -- start 2024-03-19T01:24:00.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.587+0000 7fb68429e640 1 -- start start 2024-03-19T01:24:00.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68429e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb67c0b5680 0x7fb67c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68429e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67c06de00 con 0x7fb67c0b0470 2024-03-19T01:24:00.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68429e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67c0af2f0 con 0x7fb67c072c50 2024-03-19T01:24:00.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68429e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fb67c0b5a60 con 0x7fb67c0b5680 2024-03-19T01:24:00.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68203a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb67c0b5680 0x7fb67c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68203a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb67c0b5680 0x7fb67c06e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49794/0 (socket says 172.21.15.7:49794) 2024-03-19T01:24:00.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68203a640 1 -- 172.21.15.7:0/3456139724 learned_addr learned my addr 172.21.15.7:0/3456139724 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:00.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68283b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb67c0b0470 0x7fb67c06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35258/0 (socket says 172.21.15.7:35258) 2024-03-19T01:24:00.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68203a640 1 -- 172.21.15.7:0/3456139724 >> v1:172.21.15.167:6789/0 conn(0x7fb67c072c50 legacy=0x7fb67c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:00.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68203a640 1 -- 172.21.15.7:0/3456139724 >> v1:172.21.15.7:6789/0 conn(0x7fb67c0b0470 legacy=0x7fb67c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb681038640 1 -- 172.21.15.7:0/3456139724 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1293543622 0 0) 0x7fb67c06de00 con 0x7fb67c0b0470 2024-03-19T01:24:00.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68203a640 1 -- 172.21.15.7:0/3456139724 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c0afb50 con 0x7fb67c0b5680 2024-03-19T01:24:00.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb68203a640 1 --2- 172.21.15.7:0/3456139724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b5680 0x7fb67c06e860 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb678009870 tx=0x7fb67801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4b432e4525ab73 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:00.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb681038640 1 -- 172.21.15.7:0/3456139724 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6780b1070 con 0x7fb67c0b5680 2024-03-19T01:24:00.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb681038640 1 -- 172.21.15.7:0/3456139724 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6780ac630 con 0x7fb67c0b5680 2024-03-19T01:24:00.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.591+0000 7fb681038640 1 -- 172.21.15.7:0/3456139724 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6780a8050 con 0x7fb67c0b5680 2024-03-19T01:24:00.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- 172.21.15.7:0/3456139724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b5680 msgr2=0x7fb67c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 --2- 172.21.15.7:0/3456139724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b5680 0x7fb67c06e860 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb678009870 tx=0x7fb67801c3d0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- 172.21.15.7:0/3456139724 shutdown_connections 2024-03-19T01:24:00.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 --2- 172.21.15.7:0/3456139724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b5680 0x7fb67c06e860 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- 172.21.15.7:0/3456139724 >> 172.21.15.7:0/3456139724 conn(0x7fb67c0b6510 msgr2=0x7fb67c0b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:00.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- 172.21.15.7:0/3456139724 shutdown_connections 2024-03-19T01:24:00.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- 172.21.15.7:0/3456139724 wait complete. 2024-03-19T01:24:00.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 Processor -- start 2024-03-19T01:24:00.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- start start 2024-03-19T01:24:00.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb67c072c50 0x7fb67c1e3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b0470 0x7fb67c140cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb67c0b5680 0x7fb67c14f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c1478c0 con 0x7fb67c0b0470 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c06eda0 con 0x7fb67c0b5680 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb67c147a40 con 0x7fb67c072c50 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb681839640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b0470 0x7fb67c140cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb681839640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b0470 0x7fb67c140cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49800/0 (socket says 172.21.15.7:49800) 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb681839640 1 -- 172.21.15.7:0/794554744 learned_addr learned my addr 172.21.15.7:0/794554744 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68203a640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb67c072c50 0x7fb67c1e3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68283b640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb67c0b5680 0x7fb67c14f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb681839640 1 -- 172.21.15.7:0/794554744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb67c072c50 msgr2=0x7fb67c1e3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb681839640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb67c072c50 0x7fb67c1e3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb681839640 1 -- 172.21.15.7:0/794554744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb67c0b5680 msgr2=0x7fb67c14f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb681839640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb67c0b5680 0x7fb67c14f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb681839640 1 -- 172.21.15.7:0/794554744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c141200 con 0x7fb67c0b0470 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68283b640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb67c0b5680 0x7fb67c14f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb681839640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b0470 0x7fb67c140cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00c400 tx=0x7fb66c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb672ffd640 1 -- 172.21.15.7:0/794554744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c007020 con 0x7fb67c0b0470 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb672ffd640 1 -- 172.21.15.7:0/794554744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb66c0071c0 con 0x7fb67c0b0470 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67c142ac0 con 0x7fb67c0b0470 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb672ffd640 1 -- 172.21.15.7:0/794554744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c0075b0 con 0x7fb67c0b0470 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.595+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67c1472c0 con 0x7fb67c0b0470 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.599+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67c075460 con 0x7fb67c0b0470 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.599+0000 7fb672ffd640 1 -- 172.21.15.7:0/794554744 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb66c019020 con 0x7fb67c0b0470 2024-03-19T01:24:00.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.599+0000 7fb672ffd640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb660067660 0x7fb660069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.599+0000 7fb672ffd640 1 -- 172.21.15.7:0/794554744 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb66c085010 con 0x7fb67c0b0470 2024-03-19T01:24:00.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.599+0000 7fb68203a640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb660067660 0x7fb660069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.599+0000 7fb68203a640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb660067660 0x7fb660069b20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb6780ae000 tx=0x7fb6780a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:00.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.603+0000 7fb672ffd640 1 -- 172.21.15.7:0/794554744 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb66c055d20 con 0x7fb67c0b0470 2024-03-19T01:24:00.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.799+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb67c09ee80 con 0x7fb67c0b0470 2024-03-19T01:24:00.801 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-19T01:24:00.799+0000 7fd4c7382640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T01:24:00.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.803+0000 7fb672ffd640 1 -- 172.21.15.7:0/794554744 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7fb66c055620 con 0x7fb67c0b0470 2024-03-19T01:24:00.803 INFO:teuthology.orchestra.run.smithi007.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-19T01:24:00.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb660067660 msgr2=0x7fb660069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb660067660 0x7fb660069b20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb6780ae000 tx=0x7fb6780a7000 comp rx=0 tx=0).stop 2024-03-19T01:24:00.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b0470 msgr2=0x7fb67c140cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b0470 0x7fb67c140cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00c400 tx=0x7fb66c00c8c0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 shutdown_connections 2024-03-19T01:24:00.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb660067660 0x7fb660069b20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb67c0b5680 0x7fb67c14f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb67c0b0470 0x7fb67c140cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 --2- 172.21.15.7:0/794554744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb67c072c50 0x7fb67c1e3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 >> 172.21.15.7:0/794554744 conn(0x7fb67c0b6510 msgr2=0x7fb67c06d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:00.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 shutdown_connections 2024-03-19T01:24:00.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.807+0000 7fb68429e640 1 -- 172.21.15.7:0/794554744 wait complete. 2024-03-19T01:24:00.811 INFO:tasks.ceph.mds.a.smithi007.stderr: -1344> 2024-03-19T01:23:30.380+0000 7f10c3e2d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T01:24:00.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-19T01:24:00.840 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2024-03-19T01:24:00.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14c8b5640 1 Processor -- start 2024-03-19T01:24:00.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14c8b5640 1 -- start start 2024-03-19T01:24:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14c8b5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd148074390 0x7fd1480b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14c8b5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fd1480ae250 0x7fd1480ae810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14c8b5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481445d0 0x7fd148144a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14c8b5640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fd1480afb20 con 0x7fd148074390 2024-03-19T01:24:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14c8b5640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fd148074770 con 0x7fd1480ae250 2024-03-19T01:24:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14c8b5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1480add60 con 0x7fd1481445d0 2024-03-19T01:24:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd148074390 0x7fd1480b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd148074390 0x7fd1480b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49822/0 (socket says 172.21.15.7:49822) 2024-03-19T01:24:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14659c640 1 -- 172.21.15.7:0/2358522459 learned_addr learned my addr 172.21.15.7:0/2358522459 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd145d9b640 1 --2- 172.21.15.7:0/2358522459 >> v2:172.21.15.167:3300/0 conn(0x7fd1480ae250 0x7fd1480ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14659c640 1 -- 172.21.15.7:0/2358522459 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481445d0 msgr2=0x7fd148144a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd146d9d640 1 --2- 172.21.15.7:0/2358522459 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481445d0 0x7fd148144a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14659c640 1 --2- 172.21.15.7:0/2358522459 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481445d0 0x7fd148144a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14659c640 1 -- 172.21.15.7:0/2358522459 >> v2:172.21.15.167:3300/0 conn(0x7fd1480ae250 msgr2=0x7fd1480ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14659c640 1 --2- 172.21.15.7:0/2358522459 >> v2:172.21.15.167:3300/0 conn(0x7fd1480ae250 0x7fd1480ae810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd14659c640 1 -- 172.21.15.7:0/2358522459 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1480b3180 con 0x7fd148074390 2024-03-19T01:24:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd145d9b640 1 --2- 172.21.15.7:0/2358522459 >> v2:172.21.15.167:3300/0 conn(0x7fd1480ae250 0x7fd1480ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:00.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.951+0000 7fd146d9d640 1 --2- 172.21.15.7:0/2358522459 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481445d0 0x7fd148144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:00.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14659c640 1 --2- 172.21.15.7:0/2358522459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 0x7fd1480b5140 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd1480b6a10 tx=0x7fd14001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b309c837c63dac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:00.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14559a640 1 -- 172.21.15.7:0/2358522459 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1400a55e0 con 0x7fd148074390 2024-03-19T01:24:00.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14559a640 1 -- 172.21.15.7:0/2358522459 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1400a5780 con 0x7fd148074390 2024-03-19T01:24:00.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14559a640 1 -- 172.21.15.7:0/2358522459 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1400a5b50 con 0x7fd148074390 2024-03-19T01:24:00.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 -- 172.21.15.7:0/2358522459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 msgr2=0x7fd1480b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/2358522459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 0x7fd1480b5140 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd1480b6a10 tx=0x7fd14001c960 comp rx=0 tx=0).stop 2024-03-19T01:24:00.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 -- 172.21.15.7:0/2358522459 shutdown_connections 2024-03-19T01:24:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/2358522459 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481445d0 0x7fd148144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/2358522459 >> v2:172.21.15.167:3300/0 conn(0x7fd1480ae250 0x7fd1480ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/2358522459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 0x7fd1480b5140 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 -- 172.21.15.7:0/2358522459 >> 172.21.15.7:0/2358522459 conn(0x7fd1480b5c90 msgr2=0x7fd1480b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 -- 172.21.15.7:0/2358522459 shutdown_connections 2024-03-19T01:24:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 -- 172.21.15.7:0/2358522459 wait complete. 2024-03-19T01:24:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 Processor -- start 2024-03-19T01:24:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.955+0000 7fd14c8b5640 1 -- start start 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14c8b5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 0x7fd148142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14c8b5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1481445d0 0x7fd1481e33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14c8b5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481e38e0 0x7fd148136b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14c8b5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd1480b4520 con 0x7fd148074390 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14c8b5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd1480b43a0 con 0x7fd1481445d0 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14c8b5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1480b46a0 con 0x7fd1481e38e0 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 0x7fd148142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 0x7fd148142470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49838/0 (socket says 172.21.15.7:49838) 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14659c640 1 -- 172.21.15.7:0/3282028842 learned_addr learned my addr 172.21.15.7:0/3282028842 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd145d9b640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1481445d0 0x7fd1481e33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd146d9d640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481e38e0 0x7fd148136b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14659c640 1 -- 172.21.15.7:0/3282028842 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481e38e0 msgr2=0x7fd148136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14659c640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481e38e0 0x7fd148136b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14659c640 1 -- 172.21.15.7:0/3282028842 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1481445d0 msgr2=0x7fd1481e33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14659c640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1481445d0 0x7fd1481e33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14659c640 1 -- 172.21.15.7:0/3282028842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1481458a0 con 0x7fd148074390 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd145d9b640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1481445d0 0x7fd1481e33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14659c640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 0x7fd148142470 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd1400af040 tx=0x7fd140019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd136ffd640 1 -- 172.21.15.7:0/3282028842 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1400bc020 con 0x7fd148074390 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd136ffd640 1 -- 172.21.15.7:0/3282028842 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1400b8670 con 0x7fd148074390 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd148142c70 con 0x7fd148074390 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd136ffd640 1 -- 172.21.15.7:0/3282028842 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1400b8a60 con 0x7fd148074390 2024-03-19T01:24:00.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1481e2e80 con 0x7fd148074390 2024-03-19T01:24:00.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1481e3cf0 con 0x7fd148074390 2024-03-19T01:24:00.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd136ffd640 1 -- 172.21.15.7:0/3282028842 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd1400bf020 con 0x7fd148074390 2024-03-19T01:24:00.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd136ffd640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd128067660 0x7fd128069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:00.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd136ffd640 1 -- 172.21.15.7:0/3282028842 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7fd140120ae0 con 0x7fd148074390 2024-03-19T01:24:00.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.959+0000 7fd145d9b640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd128067660 0x7fd128069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:00.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.963+0000 7fd145d9b640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd128067660 0x7fd128069b20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd13c006fd0 tx=0x7fd13c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:00.967+0000 7fd136ffd640 1 -- 172.21.15.7:0/3282028842 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd1481e3cf0 con 0x7fd148074390 2024-03-19T01:24:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.163+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd148075570 con 0x7fd148074390 2024-03-19T01:24:01.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.807+0000 7fd136ffd640 1 -- 172.21.15.7:0/3282028842 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7fd1400c6030 con 0x7fd148074390 2024-03-19T01:24:01.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd128067660 msgr2=0x7fd128069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:01.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd128067660 0x7fd128069b20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd13c006fd0 tx=0x7fd13c008040 comp rx=0 tx=0).stop 2024-03-19T01:24:01.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 msgr2=0x7fd148142470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:01.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 0x7fd148142470 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd1400af040 tx=0x7fd140019700 comp rx=0 tx=0).stop 2024-03-19T01:24:01.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 shutdown_connections 2024-03-19T01:24:01.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd128067660 0x7fd128069b20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:01.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1481e38e0 0x7fd148136b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:01.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1481445d0 0x7fd1481e33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:01.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 --2- 172.21.15.7:0/3282028842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148074390 0x7fd148142470 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:01.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 >> 172.21.15.7:0/3282028842 conn(0x7fd1480b5c90 msgr2=0x7fd1480b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:01.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 shutdown_connections 2024-03-19T01:24:01.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:01.811+0000 7fd14c8b5640 1 -- 172.21.15.7:0/3282028842 wait complete. 2024-03-19T01:24:01.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-19T01:24:02.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdca4eac640 1 Processor -- start 2024-03-19T01:24:02.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdca4eac640 1 -- start start 2024-03-19T01:24:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdca4eac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdca006c2f0 0x7fdca00ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdca4eac640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fdca014acc0 0x7fdca014b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdca4eac640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdca006d6d0 con 0x7fdca014b660 2024-03-19T01:24:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdca4eac640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fdca00b2fd0 con 0x7fdca014acc0 2024-03-19T01:24:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdca4eac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdca00af4e0 con 0x7fdca006c2f0 2024-03-19T01:24:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdc9ed9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdca014b660 0x7fdca00b3470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35268/0 (socket says 172.21.15.7:35268) 2024-03-19T01:24:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdc9ed9d640 1 -- 172.21.15.7:0/2526660617 learned_addr learned my addr 172.21.15.7:0/2526660617 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdc9dd9b640 1 --2- 172.21.15.7:0/2526660617 >> v2:172.21.15.167:3300/0 conn(0x7fdca014acc0 0x7fdca014b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdc9e59c640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdca006c2f0 0x7fdca00ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468551249 0 0) 0x7fdca006d6d0 con 0x7fdca014b660 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.015+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc8c003530 con 0x7fdca014b660 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3419122152 0 0) 0x7fdc8c003530 con 0x7fdca014b660 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdca006c2f0 msgr2=0x7fdca00ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdca006c2f0 0x7fdca00ad9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 >> v2:172.21.15.167:3300/0 conn(0x7fdca014acc0 msgr2=0x7fdca014b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 --2- 172.21.15.7:0/2526660617 >> v2:172.21.15.167:3300/0 conn(0x7fdca014acc0 0x7fdca014b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca014bad0 con 0x7fdca014b660 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fdc90001a00 con 0x7fdca014b660 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 >> v1:172.21.15.7:6789/0 conn(0x7fdca014b660 legacy=0x7fdca00b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0039c0 0x7fdc8c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc8c004810 0x7fdc8c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdca006c2f0 0x7fdc8c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdca00af4e0 con 0x7fdc8c004810 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c00b9d0 con 0x7fdc8c0039c0 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdc8c00bb50 con 0x7fdca006c2f0 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9dd9b640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc8c004810 0x7fdc8c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9e59c640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0039c0 0x7fdc8c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9ed9d640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdca006c2f0 0x7fdc8c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9dd9b640 1 -- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdca006c2f0 msgr2=0x7fdc8c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9dd9b640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdca006c2f0 0x7fdc8c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9dd9b640 1 -- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0039c0 msgr2=0x7fdc8c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9dd9b640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0039c0 0x7fdc8c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9dd9b640 1 -- 172.21.15.7:0/2526660617 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8c003530 con 0x7fdc8c004810 2024-03-19T01:24:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9e59c640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0039c0 0x7fdc8c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9dd9b640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc8c004810 0x7fdc8c004c90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009ae0 tx=0x7fdc8801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af33c10220ff3f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc880096b0 con 0x7fdc8c004810 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc880a64c0 con 0x7fdc8c004810 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9d59a640 1 -- 172.21.15.7:0/2526660617 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc880a68b0 con 0x7fdc8c004810 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdca4eac640 1 -- 172.21.15.7:0/2526660617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc8c004810 msgr2=0x7fdc8c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdca4eac640 1 --2- 172.21.15.7:0/2526660617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc8c004810 0x7fdc8c004c90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009ae0 tx=0x7fdc8801ca50 comp rx=0 tx=0).stop 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdc9ed9d640 1 -- 172.21.15.7:0/2526660617 reap_dead start 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdca4eac640 1 -- 172.21.15.7:0/2526660617 shutdown_connections 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.019+0000 7fdca4eac640 1 -- 172.21.15.7:0/2526660617 >> 172.21.15.7:0/2526660617 conn(0x7fdca00b8c40 msgr2=0x7fdca00b9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 -- 172.21.15.7:0/2526660617 shutdown_connections 2024-03-19T01:24:02.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 -- 172.21.15.7:0/2526660617 wait complete. 2024-03-19T01:24:02.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 Processor -- start 2024-03-19T01:24:02.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 -- start start 2024-03-19T01:24:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c004810 0x7fdca01dfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca0148790 0x7fdca0148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca0149130 0x7fdca01495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdca00afb10 con 0x7fdca0148790 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdca00af200 con 0x7fdca0149130 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdca00b5020 con 0x7fdc8c004810 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca0148790 0x7fdca0148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca0148790 0x7fdca0148bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49856/0 (socket says 172.21.15.7:49856) 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9dd9b640 1 -- 172.21.15.7:0/1571276200 learned_addr learned my addr 172.21.15.7:0/1571276200 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9e59c640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c004810 0x7fdca01dfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9ed9d640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca0149130 0x7fdca01495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9dd9b640 1 -- 172.21.15.7:0/1571276200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c004810 msgr2=0x7fdca01dfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9dd9b640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c004810 0x7fdca01dfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9dd9b640 1 -- 172.21.15.7:0/1571276200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca0149130 msgr2=0x7fdca01495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9dd9b640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca0149130 0x7fdca01495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9dd9b640 1 -- 172.21.15.7:0/1571276200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca01d2eb0 con 0x7fdca0148790 2024-03-19T01:24:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9ed9d640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca0149130 0x7fdca01495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc9dd9b640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca0148790 0x7fdca0148bf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdc880982d0 tx=0x7fdc8c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc96ffd640 1 -- 172.21.15.7:0/1571276200 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc880bd020 con 0x7fdca0148790 2024-03-19T01:24:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc96ffd640 1 -- 172.21.15.7:0/1571276200 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc880a6880 con 0x7fdca0148790 2024-03-19T01:24:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca0149af0 con 0x7fdca0148790 2024-03-19T01:24:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdc96ffd640 1 -- 172.21.15.7:0/1571276200 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc88005e60 con 0x7fdca0148790 2024-03-19T01:24:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca01d3c10 con 0x7fdca0148790 2024-03-19T01:24:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.023+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca0075570 con 0x7fdca0148790 2024-03-19T01:24:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.027+0000 7fdc96ffd640 1 -- 172.21.15.7:0/1571276200 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdc880c0020 con 0x7fdca0148790 2024-03-19T01:24:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.027+0000 7fdc96ffd640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdc80067660 0x7fdc80069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.027+0000 7fdc96ffd640 1 -- 172.21.15.7:0/1571276200 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7fdc88120560 con 0x7fdca0148790 2024-03-19T01:24:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.027+0000 7fdc9e59c640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdc80067660 0x7fdc80069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.027+0000 7fdc9e59c640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdc80067660 0x7fdc80069b20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009a90 tx=0x7fdc98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:02.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.031+0000 7fdc96ffd640 1 -- 172.21.15.7:0/1571276200 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdc880c7030 con 0x7fdca0148790 2024-03-19T01:24:02.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.191+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdca006b9c0 con 0x7fdca0148790 2024-03-19T01:24:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.811+0000 7fdc96ffd640 1 -- 172.21.15.7:0/1571276200 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v23) ==== 181+0+0 (secure 0 0 0) 0x7fdca01d2eb0 con 0x7fdca0148790 2024-03-19T01:24:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' removed 2024-03-19T01:24:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdc80067660 msgr2=0x7fdc80069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdc80067660 0x7fdc80069b20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009a90 tx=0x7fdc98008040 comp rx=0 tx=0).stop 2024-03-19T01:24:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca0148790 msgr2=0x7fdca0148bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca0148790 0x7fdca0148bf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdc880982d0 tx=0x7fdc8c005840 comp rx=0 tx=0).stop 2024-03-19T01:24:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 shutdown_connections 2024-03-19T01:24:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fdc80067660 0x7fdc80069b20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca0149130 0x7fdca01495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca0148790 0x7fdca0148bf0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 --2- 172.21.15.7:0/1571276200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c004810 0x7fdca01dfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 >> 172.21.15.7:0/1571276200 conn(0x7fdca00b8c40 msgr2=0x7fdca014a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 shutdown_connections 2024-03-19T01:24:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.815+0000 7fdca4eac640 1 -- 172.21.15.7:0/1571276200 wait complete. 2024-03-19T01:24:02.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-19T01:24:02.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61faf2640 1 Processor -- start 2024-03-19T01:24:02.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61faf2640 1 -- start start 2024-03-19T01:24:02.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61faf2640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fe6180b7930 0x7fe618145370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61faf2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6181458b0 0x7fe61814fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61faf2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6180b5640 con 0x7fe6180b6ea0 2024-03-19T01:24:02.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61faf2640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fe6180b7d10 con 0x7fe6180b7930 2024-03-19T01:24:02.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61faf2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe6180b6630 con 0x7fe6181458b0 2024-03-19T01:24:02.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d08d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe6180b6ea0 0x7fe6180b45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51108/0 (socket says 172.21.15.7:51108) 2024-03-19T01:24:02.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d08d640 1 -- 172.21.15.7:0/544441406 learned_addr learned my addr 172.21.15.7:0/544441406 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:02.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d88e640 1 --2- 172.21.15.7:0/544441406 >> v2:172.21.15.167:3300/0 conn(0x7fe6180b7930 0x7fe618145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61e08f640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6181458b0 0x7fe61814fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127512942 0 0) 0x7fe6180b5640 con 0x7fe6180b6ea0 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe604003530 con 0x7fe6180b6ea0 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3592018662 0 0) 0x7fe604003530 con 0x7fe6180b6ea0 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6181458b0 msgr2=0x7fe61814fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6181458b0 0x7fe61814fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 >> v2:172.21.15.167:3300/0 conn(0x7fe6180b7930 msgr2=0x7fe618145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 --2- 172.21.15.7:0/544441406 >> v2:172.21.15.167:3300/0 conn(0x7fe6180b7930 0x7fe618145370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6180b58e0 con 0x7fe6180b6ea0 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fe608001680 con 0x7fe6180b6ea0 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 >> v1:172.21.15.7:6789/0 conn(0x7fe6180b6ea0 legacy=0x7fe6180b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6040039c0 0x7fe604003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe604004810 0x7fe604004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6181458b0 0x7fe6040051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe6180b7d10 con 0x7fe604004810 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe6180b6630 con 0x7fe6040039c0 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe60400b9d0 con 0x7fe6181458b0 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d08d640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe604004810 0x7fe604004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d88e640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6040039c0 0x7fe604003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61e08f640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6181458b0 0x7fe6040051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d08d640 1 -- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6181458b0 msgr2=0x7fe6040051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d08d640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6181458b0 0x7fe6040051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d08d640 1 -- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6040039c0 msgr2=0x7fe604003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d08d640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6040039c0 0x7fe604003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d08d640 1 -- 172.21.15.7:0/544441406 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe604003530 con 0x7fe604004810 2024-03-19T01:24:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.975+0000 7fe61d88e640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6040039c0 0x7fe604003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:02.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61d08d640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe604004810 0x7fe604004c90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c160 tx=0x7fe60801f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2fa7959c88294bc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe608007c90 con 0x7fe604004810 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6080ac020 con 0x7fe604004810 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61c88c640 1 -- 172.21.15.7:0/544441406 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6080058e0 con 0x7fe604004810 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61faf2640 1 -- 172.21.15.7:0/544441406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe604004810 msgr2=0x7fe604004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61faf2640 1 --2- 172.21.15.7:0/544441406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe604004810 0x7fe604004c90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c160 tx=0x7fe60801f2d0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61e08f640 1 -- 172.21.15.7:0/544441406 reap_dead start 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61faf2640 1 -- 172.21.15.7:0/544441406 shutdown_connections 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61faf2640 1 -- 172.21.15.7:0/544441406 >> 172.21.15.7:0/544441406 conn(0x7fe61806b2c0 msgr2=0x7fe61806b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61faf2640 1 -- 172.21.15.7:0/544441406 shutdown_connections 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61faf2640 1 -- 172.21.15.7:0/544441406 wait complete. 2024-03-19T01:24:02.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61faf2640 1 Processor -- start 2024-03-19T01:24:02.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.979+0000 7fe61faf2640 1 -- start start 2024-03-19T01:24:02.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61faf2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe604004810 0x7fe6181ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61faf2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6181cbe30 0x7fe61813aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61faf2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe61813b430 0x7fe6181dad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61faf2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe618072b30 con 0x7fe6181cbe30 2024-03-19T01:24:02.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61faf2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe6180b4d00 con 0x7fe604004810 2024-03-19T01:24:02.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61faf2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe618072cb0 con 0x7fe61813b430 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d08d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6181cbe30 0x7fe61813aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d08d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6181cbe30 0x7fe61813aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48656/0 (socket says 172.21.15.7:48656) 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d08d640 1 -- 172.21.15.7:0/1311712626 learned_addr learned my addr 172.21.15.7:0/1311712626 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61e08f640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe61813b430 0x7fe6181dad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d88e640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe604004810 0x7fe6181ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d08d640 1 -- 172.21.15.7:0/1311712626 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe61813b430 msgr2=0x7fe6181dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d08d640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe61813b430 0x7fe6181dad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d08d640 1 -- 172.21.15.7:0/1311712626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe604004810 msgr2=0x7fe6181ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d08d640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe604004810 0x7fe6181ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d08d640 1 -- 172.21.15.7:0/1311712626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6181ce780 con 0x7fe6181cbe30 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d08d640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6181cbe30 0x7fe61813aef0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe6080ae040 tx=0x7fe604005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe60e7fc640 1 -- 172.21.15.7:0/1311712626 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6080047f0 con 0x7fe6181cbe30 2024-03-19T01:24:02.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe60e7fc640 1 -- 172.21.15.7:0/1311712626 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6080b6070 con 0x7fe6181cbe30 2024-03-19T01:24:02.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe60e7fc640 1 -- 172.21.15.7:0/1311712626 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe608004b90 con 0x7fe6181cbe30 2024-03-19T01:24:02.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6181cd570 con 0x7fe6181cbe30 2024-03-19T01:24:02.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6181cc290 con 0x7fe6181cbe30 2024-03-19T01:24:02.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe618075570 con 0x7fe6181cbe30 2024-03-19T01:24:02.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe60e7fc640 1 -- 172.21.15.7:0/1311712626 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe6080070d0 con 0x7fe6181cbe30 2024-03-19T01:24:02.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe60e7fc640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe5fc067690 0x7fe5fc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:02.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d88e640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe5fc067690 0x7fe5fc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:02.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe60e7fc640 1 -- 172.21.15.7:0/1311712626 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5357+0+0 (secure 0 0 0) 0x7fe6081225d0 con 0x7fe6181cbe30 2024-03-19T01:24:02.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.983+0000 7fe61d88e640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe5fc067690 0x7fe5fc069b50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe610007440 tx=0x7fe61000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:02.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:02.991+0000 7fe60e7fc640 1 -- 172.21.15.7:0/1311712626 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe6181ce780 con 0x7fe6181cbe30 2024-03-19T01:24:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.147+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe61813b810 con 0x7fe6181cbe30 2024-03-19T01:24:03.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.819+0000 7fe60e7fc640 1 -- 172.21.15.7:0/1311712626 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v24) ==== 169+0+0 (secure 0 0 0) 0x7fe608019040 con 0x7fe6181cbe30 2024-03-19T01:24:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' removed 2024-03-19T01:24:03.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe5fc067690 msgr2=0x7fe5fc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:03.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe5fc067690 0x7fe5fc069b50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe610007440 tx=0x7fe61000b040 comp rx=0 tx=0).stop 2024-03-19T01:24:03.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6181cbe30 msgr2=0x7fe61813aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6181cbe30 0x7fe61813aef0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe6080ae040 tx=0x7fe604005840 comp rx=0 tx=0).stop 2024-03-19T01:24:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 shutdown_connections 2024-03-19T01:24:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe5fc067690 0x7fe5fc069b50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe61813b430 0x7fe6181dad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6181cbe30 0x7fe61813aef0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 --2- 172.21.15.7:0/1311712626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe604004810 0x7fe6181ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 >> 172.21.15.7:0/1311712626 conn(0x7fe61806b2c0 msgr2=0x7fe6181432e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:03.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 shutdown_connections 2024-03-19T01:24:03.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:03.823+0000 7fe61faf2640 1 -- 172.21.15.7:0/1311712626 wait complete. 2024-03-19T01:24:03.850 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-19T01:24:03.850 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-19T01:24:03.850 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-19T01:24:03.850 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:24:03.850 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-19T01:24:03.851 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-19T01:24:03.851 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-19T01:24:03.851 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:24:03.862 INFO:tasks.ceph.mds.a:Stopped 2024-03-19T01:24:03.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-19T01:24:03.869 INFO:tasks.ceph.mds.b:Stopped 2024-03-19T01:24:03.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-19T01:24:03.909 INFO:tasks.ceph.mds.a:Started 2024-03-19T01:24:03.909 DEBUG:teuthology.parallel:result is None 2024-03-19T01:24:03.914 INFO:tasks.ceph.mds.b:Started 2024-03-19T01:24:03.914 DEBUG:teuthology.parallel:result is None 2024-03-19T01:24:03.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-19T01:24:03.981 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2024-03-19T01:24:03.989 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-03-19T01:24:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b95ef640 1 Processor -- start 2024-03-19T01:24:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b95ef640 1 -- start start 2024-03-19T01:24:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b95ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b40bd660 0x7fd5b414a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:04.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b95ef640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd5b414ae60 0x7fd5b4151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b95ef640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b40b6ce0 con 0x7fd5b40b4570 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b95ef640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b40b4950 con 0x7fd5b414ae60 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b95ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b40b51c0 con 0x7fd5b40bd660 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b37fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd5b414ae60 0x7fd5b4151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b2ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd5b40b4570 0x7fd5b4140fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51112/0 (socket says 172.21.15.7:51112) 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b2ffd640 1 -- 172.21.15.7:0/1282191772 learned_addr learned my addr 172.21.15.7:0/1282191772 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b27fc640 1 --2- 172.21.15.7:0/1282191772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b40bd660 0x7fd5b414a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b1ffb640 1 -- 172.21.15.7:0/1282191772 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034981196 0 0) 0x7fd5b40b6ce0 con 0x7fd5b40b4570 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b1ffb640 1 -- 172.21.15.7:0/1282191772 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd59c003530 con 0x7fd5b40b4570 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b37fe640 1 -- 172.21.15.7:0/1282191772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b40bd660 msgr2=0x7fd5b414a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b37fe640 1 --2- 172.21.15.7:0/1282191772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b40bd660 0x7fd5b414a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b37fe640 1 -- 172.21.15.7:0/1282191772 >> v1:172.21.15.7:6789/0 conn(0x7fd5b40b4570 legacy=0x7fd5b4140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b37fe640 1 -- 172.21.15.7:0/1282191772 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b40b5480 con 0x7fd5b414ae60 2024-03-19T01:24:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b37fe640 1 --2- 172.21.15.7:0/1282191772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b414ae60 0x7fd5b4151cb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009d90 tx=0x7fd5ac01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43fdf0e499dabc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b1ffb640 1 -- 172.21.15.7:0/1282191772 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac0b2070 con 0x7fd5b414ae60 2024-03-19T01:24:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.043+0000 7fd5b1ffb640 1 -- 172.21.15.7:0/1282191772 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5ac0a5710 con 0x7fd5b414ae60 2024-03-19T01:24:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b1ffb640 1 -- 172.21.15.7:0/1282191772 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac0a5ae0 con 0x7fd5b414ae60 2024-03-19T01:24:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1282191772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b414ae60 msgr2=0x7fd5b4151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 --2- 172.21.15.7:0/1282191772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b414ae60 0x7fd5b4151cb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009d90 tx=0x7fd5ac01c850 comp rx=0 tx=0).stop 2024-03-19T01:24:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1282191772 shutdown_connections 2024-03-19T01:24:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 --2- 172.21.15.7:0/1282191772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b414ae60 0x7fd5b4151cb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 --2- 172.21.15.7:0/1282191772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b40bd660 0x7fd5b414a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1282191772 >> 172.21.15.7:0/1282191772 conn(0x7fd5b4072df0 msgr2=0x7fd5b4073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1282191772 shutdown_connections 2024-03-19T01:24:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1282191772 wait complete. 2024-03-19T01:24:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 Processor -- start 2024-03-19T01:24:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- start start 2024-03-19T01:24:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b40b4570 0x7fd5b41395b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b414ae60 0x7fd5b4141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b4142330 0x7fd5b4142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b40b93b0 con 0x7fd5b40b4570 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b40b9230 con 0x7fd5b4142330 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b4154a10 con 0x7fd5b414ae60 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b2ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b40b4570 0x7fd5b41395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b2ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b40b4570 0x7fd5b41395b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48706/0 (socket says 172.21.15.7:48706) 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b2ffd640 1 -- 172.21.15.7:0/1521799961 learned_addr learned my addr 172.21.15.7:0/1521799961 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b27fc640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b414ae60 0x7fd5b4141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b37fe640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b4142330 0x7fd5b4142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b2ffd640 1 -- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b414ae60 msgr2=0x7fd5b4141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b2ffd640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b414ae60 0x7fd5b4141df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b2ffd640 1 -- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b4142330 msgr2=0x7fd5b4142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b2ffd640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b4142330 0x7fd5b4142ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b2ffd640 1 -- 172.21.15.7:0/1521799961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b413ba30 con 0x7fd5b40b4570 2024-03-19T01:24:04.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b27fc640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b414ae60 0x7fd5b4141df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b37fe640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b4142330 0x7fd5b4142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b2ffd640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b40b4570 0x7fd5b41395b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c480 tx=0x7fd5a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd59b7fe640 1 -- 172.21.15.7:0/1521799961 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4017070 con 0x7fd5b40b4570 2024-03-19T01:24:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd59b7fe640 1 -- 172.21.15.7:0/1521799961 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a4005cf0 con 0x7fd5b40b4570 2024-03-19T01:24:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b413a180 con 0x7fd5b40b4570 2024-03-19T01:24:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd59b7fe640 1 -- 172.21.15.7:0/1521799961 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4007290 con 0x7fd5b40b4570 2024-03-19T01:24:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.047+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b4144a90 con 0x7fd5b40b4570 2024-03-19T01:24:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.051+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b41445c0 con 0x7fd5b40b4570 2024-03-19T01:24:04.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.051+0000 7fd59b7fe640 1 -- 172.21.15.7:0/1521799961 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd5a401a070 con 0x7fd5b40b4570 2024-03-19T01:24:04.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.051+0000 7fd59b7fe640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd5a0067660 0x7fd5a0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:04.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.051+0000 7fd59b7fe640 1 -- 172.21.15.7:0/1521799961 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 4943+0+0 (secure 0 0 0) 0x7fd5a4084970 con 0x7fd5b40b4570 2024-03-19T01:24:04.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.051+0000 7fd5b27fc640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd5a0067660 0x7fd5a0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:04.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.051+0000 7fd5b27fc640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd5a0067660 0x7fd5a0069b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd5b41519a0 tx=0x7fd5a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:04.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.055+0000 7fd59b7fe640 1 -- 172.21.15.7:0/1521799961 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd5a4055980 con 0x7fd5b40b4570 2024-03-19T01:24:04.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.235+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fd5b4075460 con 0x7fd5b40b4570 2024-03-19T01:24:04.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.839+0000 7fd59b7fe640 1 -- 172.21.15.7:0/1521799961 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v26) ==== 97+0+0 (secure 0 0 0) 0x7fd5a4055280 con 0x7fd5b40b4570 2024-03-19T01:24:04.842 INFO:teuthology.orchestra.run.smithi007.stderr: removed all blocklist entries 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd5a0067660 msgr2=0x7fd5a0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd5a0067660 0x7fd5a0069b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd5b41519a0 tx=0x7fd5a8008040 comp rx=0 tx=0).stop 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b40b4570 msgr2=0x7fd5b41395b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b40b4570 0x7fd5b41395b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c480 tx=0x7fd5a400c940 comp rx=0 tx=0).stop 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 shutdown_connections 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd5a0067660 0x7fd5a0069b20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5b414ae60 0x7fd5b4141df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5b4142330 0x7fd5b4142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 --2- 172.21.15.7:0/1521799961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5b40b4570 0x7fd5b41395b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 >> 172.21.15.7:0/1521799961 conn(0x7fd5b4072df0 msgr2=0x7fd5b4147360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 shutdown_connections 2024-03-19T01:24:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.843+0000 7fd5b95ef640 1 -- 172.21.15.7:0/1521799961 wait complete. 2024-03-19T01:24:04.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-19T01:24:04.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bf992e640 1 Processor -- start 2024-03-19T01:24:04.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bf992e640 1 -- start start 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bf992e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf40b1460 0x7f8bf40b6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bf992e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8bf406d870 0x7f8bf4144700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bf992e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f8bf40b80f0 0x7f8bf4144c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bf992e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f8bf40b1840 con 0x7f8bf406d870 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bf992e640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f8bf40b2120 con 0x7f8bf40b80f0 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bf992e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf41441e0 con 0x7f8bf40b1460 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bebfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8bf406d870 0x7f8bf4144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bebfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8bf406d870 0x7f8bf4144700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48722/0 (socket says 172.21.15.7:48722) 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bebfff640 1 -- 172.21.15.7:0/1662492031 learned_addr learned my addr 172.21.15.7:0/1662492031 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bf2ffd640 1 --2- 172.21.15.7:0/1662492031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf40b1460 0x7f8bf40b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:04.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bebfff640 1 -- 172.21.15.7:0/1662492031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf40b1460 msgr2=0x7f8bf40b6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:04.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bebfff640 1 --2- 172.21.15.7:0/1662492031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf40b1460 0x7f8bf40b6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bebfff640 1 -- 172.21.15.7:0/1662492031 >> v2:172.21.15.167:3300/0 conn(0x7f8bf40b80f0 msgr2=0x7f8bf4144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:04.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bebfff640 1 --2- 172.21.15.7:0/1662492031 >> v2:172.21.15.167:3300/0 conn(0x7f8bf40b80f0 0x7f8bf4144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:04.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bebfff640 1 -- 172.21.15.7:0/1662492031 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf40b85c0 con 0x7f8bf406d870 2024-03-19T01:24:04.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.995+0000 7f8bebfff640 1 --2- 172.21.15.7:0/1662492031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf406d870 0x7f8bf4144700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4072bc0 tx=0x7f8bdc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a4921542549a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:04.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf27fc640 1 -- 172.21.15.7:0/1662492031 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc0b0070 con 0x7f8bf406d870 2024-03-19T01:24:04.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf27fc640 1 -- 172.21.15.7:0/1662492031 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bdc0ab600 con 0x7f8bf406d870 2024-03-19T01:24:04.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf27fc640 1 -- 172.21.15.7:0/1662492031 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc0ab9d0 con 0x7f8bf406d870 2024-03-19T01:24:05.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- 172.21.15.7:0/1662492031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf406d870 msgr2=0x7f8bf4144700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 --2- 172.21.15.7:0/1662492031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf406d870 0x7f8bf4144700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4072bc0 tx=0x7f8bdc01c490 comp rx=0 tx=0).stop 2024-03-19T01:24:05.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- 172.21.15.7:0/1662492031 shutdown_connections 2024-03-19T01:24:05.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 --2- 172.21.15.7:0/1662492031 >> v2:172.21.15.167:3300/0 conn(0x7f8bf40b80f0 0x7f8bf4144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 --2- 172.21.15.7:0/1662492031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf406d870 0x7f8bf4144700 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 --2- 172.21.15.7:0/1662492031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf40b1460 0x7f8bf40b6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- 172.21.15.7:0/1662492031 >> 172.21.15.7:0/1662492031 conn(0x7f8bf40b5330 msgr2=0x7f8bf40b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:05.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- 172.21.15.7:0/1662492031 shutdown_connections 2024-03-19T01:24:05.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- 172.21.15.7:0/1662492031 wait complete. 2024-03-19T01:24:05.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 Processor -- start 2024-03-19T01:24:05.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- start start 2024-03-19T01:24:05.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf406d870 0x7f8bf4143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf40b1460 0x7f8bf41e4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8bf40b80f0 0x7f8bf413a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf41481e0 con 0x7f8bf40b1460 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf4148060 con 0x7f8bf40b80f0 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf4148360 con 0x7f8bf406d870 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf40b1460 0x7f8bf41e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bebfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf40b1460 0x7f8bf41e4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48730/0 (socket says 172.21.15.7:48730) 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bebfff640 1 -- 172.21.15.7:0/422620992 learned_addr learned my addr 172.21.15.7:0/422620992 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf2ffd640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf406d870 0x7f8bf4143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bebfff640 1 -- 172.21.15.7:0/422620992 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf406d870 msgr2=0x7f8bf4143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bebfff640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf406d870 0x7f8bf4143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bebfff640 1 -- 172.21.15.7:0/422620992 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8bf40b80f0 msgr2=0x7f8bf413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bebfff640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8bf40b80f0 0x7f8bf413a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bebfff640 1 -- 172.21.15.7:0/422620992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf41455c0 con 0x7f8bf40b1460 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bebfff640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf40b1460 0x7f8bf41e4130 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc01ab40 tx=0x7f8bdc019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:05.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf0ff9640 1 -- 172.21.15.7:0/422620992 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc0b9070 con 0x7f8bf40b1460 2024-03-19T01:24:05.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf0ff9640 1 -- 172.21.15.7:0/422620992 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bdc0b4600 con 0x7f8bf40b1460 2024-03-19T01:24:05.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf4145230 con 0x7f8bf40b1460 2024-03-19T01:24:05.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf0ff9640 1 -- 172.21.15.7:0/422620992 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc0b0040 con 0x7f8bf40b1460 2024-03-19T01:24:05.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:04.999+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf41e7bd0 con 0x7f8bf40b1460 2024-03-19T01:24:05.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.003+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf4075460 con 0x7f8bf40b1460 2024-03-19T01:24:05.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.003+0000 7f8bf0ff9640 1 -- 172.21.15.7:0/422620992 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8bdc0bc070 con 0x7f8bf40b1460 2024-03-19T01:24:05.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.003+0000 7f8bf0ff9640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8bd4067660 0x7f8bd4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.003+0000 7f8bf0ff9640 1 -- 172.21.15.7:0/422620992 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f8bdc11e790 con 0x7f8bf40b1460 2024-03-19T01:24:05.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.003+0000 7f8bf2ffd640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8bd4067660 0x7f8bd4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.003+0000 7f8bf2ffd640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8bd4067660 0x7f8bd4069b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8be400b800 tx=0x7f8be4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:05.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.007+0000 7f8bf0ff9640 1 -- 172.21.15.7:0/422620992 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8bdc016110 con 0x7f8bf40b1460 2024-03-19T01:24:05.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f8bf414e770 con 0x7f8bf40b1460 2024-03-19T01:24:05.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf0ff9640 1 -- 172.21.15.7:0/422620992 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+5207 (secure 0 0 0) 0x7f8bdc0ef870 con 0x7f8bf40b1460 2024-03-19T01:24:05.228 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:05.228 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:05.228 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_dump": [ 2024-03-19T01:24:05.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.228 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "mds.a", 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDR6PhlK/AUDBAA94pHbItEQ9o86j85imfdIA==", 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": "allow", 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow *", 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow *", 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "mds.b", 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDR6PhljWJJEhAAhzDx2ziIc1Xqn2GMaQuJSw==", 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": "allow", 2024-03-19T01:24:05.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow *", 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow *", 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "osd.0", 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDS6PhlNphlKBAAyunkfFzvQpgLzXiQph4oyQ==", 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow profile osd", 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile osd", 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "osd.1", 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDV6Phli9EGLBAAWnUfdc3mT1J5vvQGjw4FAw==", 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow profile osd", 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile osd", 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "osd.2", 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDY6PhlmhQFMhAAwo0If4nQ49q643BAJK+Z+A==", 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow profile osd", 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile osd", 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "osd.3", 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDb6PhlB5a5NxAAArM2TyZLtBVtSY0i5regNg==", 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow profile osd", 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile osd", 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "osd.4", 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDh6Phl1Su8OBAAeE9KMWzAW6lIfvv1VLrOQw==", 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow profile osd", 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile osd", 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "osd.5", 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDl6PhljbpUAhAAC7o7ETDafwt1BJxzituBxQ==", 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow profile osd", 2024-03-19T01:24:05.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile osd", 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "osd.6", 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDo6Phl4qrACBAAmjBNXyYzX7mEGR0ciRoHTA==", 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow profile osd", 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile osd", 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "osd.7", 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDr6PhlpJXxDxAAB17KJ8tzGewusYgGNmg5nA==", 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow profile osd", 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile osd", 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "client.0", 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDR6Phl6xj5FxAAeVMK9nlnMPrcDkxprwQY2w==", 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": "allow", 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow r", 2024-03-19T01:24:05.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow rw", 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow rwx" 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "client.admin", 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDQ6Phl61PYMBAAK/5GIWIK8AXAbGNGjY919w==", 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": "allow *", 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": "allow *", 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow *", 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "client.bootstrap-mds", 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQD16Phlh0ZNMBAAaxp3g1yMwLXXAQPRlQiRCw==", 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile bootstrap-mds" 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "client.bootstrap-mgr", 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQD16PhlH19NMBAA7hX9mtiGT8e8z8Rixu71Aw==", 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "client.bootstrap-osd", 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQD16PhlYXdNMBAAyrlfO2QRcRs4J3xQP8QkXw==", 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile bootstrap-osd" 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "client.bootstrap-rbd", 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQD16PhlNI5NMBAA/QVauYFM7AJnrIYwYc+GeQ==", 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQD16PhlyqNNMBAA9A7kXXBaY2KAUap+mFhiPQ==", 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "client.bootstrap-rgw", 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQD16Phlf7pNMBAAdm1pLz4rYIZcuxq3Fa+pOQ==", 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "mgr.x", 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDR6PhlET0KCRAA70G3XCxdRYYJWv1Yw5u3dg==", 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": "allow *", 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile mgr", 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "entity": "mgr.y", 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "AQDR6PhlSe9fBhAAvNqkKFNocL9xCVoNfTGX0Q==", 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": { 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": "allow *", 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": "allow profile mgr", 2024-03-19T01:24:05.240 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": "allow *" 2024-03-19T01:24:05.241 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.241 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:05.241 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-19T01:24:05.241 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8bd4067660 msgr2=0x7f8bd4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8bd4067660 0x7f8bd4069b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8be400b800 tx=0x7f8be4008040 comp rx=0 tx=0).stop 2024-03-19T01:24:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf40b1460 msgr2=0x7f8bf41e4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf40b1460 0x7f8bf41e4130 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc01ab40 tx=0x7f8bdc019230 comp rx=0 tx=0).stop 2024-03-19T01:24:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 shutdown_connections 2024-03-19T01:24:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8bd4067660 0x7f8bd4069b20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8bf40b80f0 0x7f8bf413a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bf40b1460 0x7f8bf41e4130 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 --2- 172.21.15.7:0/422620992 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf406d870 0x7f8bf4143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 >> 172.21.15.7:0/422620992 conn(0x7f8bf40b5330 msgr2=0x7f8bf40b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:05.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 shutdown_connections 2024-03-19T01:24:05.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.227+0000 7f8bf992e640 1 -- 172.21.15.7:0/422620992 wait complete. 2024-03-19T01:24:05.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-19T01:24:05.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c930c640 1 Processor -- start 2024-03-19T01:24:05.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c930c640 1 -- start start 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c930c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f58c40b7a90 0x7f58c40ba0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c930c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b88c0 0x7f58c406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c930c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c40b9bd0 con 0x7f58c406e0e0 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c930c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f58c40b7e70 con 0x7f58c40b7a90 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c930c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f58c40b95d0 con 0x7f58c40b88c0 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c2ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f58c40b7a90 0x7f58c40ba0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c2ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f58c40b7a90 0x7f58c40ba0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48742/0 (socket says 172.21.15.7:48742) 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c2ffd640 1 -- 172.21.15.7:0/979435203 learned_addr learned my addr 172.21.15.7:0/979435203 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58bbfff640 1 --2- 172.21.15.7:0/979435203 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b88c0 0x7f58c406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c2ffd640 1 -- 172.21.15.7:0/979435203 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b88c0 msgr2=0x7f58c406d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c2ffd640 1 --2- 172.21.15.7:0/979435203 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b88c0 0x7f58c406d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c2ffd640 1 -- 172.21.15.7:0/979435203 >> v1:172.21.15.167:6789/0 conn(0x7f58c406e0e0 legacy=0x7f58c4141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.367+0000 7f58c2ffd640 1 -- 172.21.15.7:0/979435203 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c406d7e0 con 0x7f58c40b7a90 2024-03-19T01:24:05.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c2ffd640 1 --2- 172.21.15.7:0/979435203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b7a90 0x7f58c40ba0b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009870 tx=0x7f58ac01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c5ef036a885612 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:05.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c27fc640 1 -- 172.21.15.7:0/979435203 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac0ab680 con 0x7f58c40b7a90 2024-03-19T01:24:05.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c27fc640 1 -- 172.21.15.7:0/979435203 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58ac0b0070 con 0x7f58c40b7a90 2024-03-19T01:24:05.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c27fc640 1 -- 172.21.15.7:0/979435203 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac0aba40 con 0x7f58c40b7a90 2024-03-19T01:24:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c930c640 1 -- 172.21.15.7:0/979435203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b7a90 msgr2=0x7f58c40ba0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c930c640 1 --2- 172.21.15.7:0/979435203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b7a90 0x7f58c40ba0b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009870 tx=0x7f58ac01c560 comp rx=0 tx=0).stop 2024-03-19T01:24:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c930c640 1 -- 172.21.15.7:0/979435203 shutdown_connections 2024-03-19T01:24:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c930c640 1 --2- 172.21.15.7:0/979435203 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b88c0 0x7f58c406d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c930c640 1 --2- 172.21.15.7:0/979435203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b7a90 0x7f58c40ba0b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c930c640 1 -- 172.21.15.7:0/979435203 >> 172.21.15.7:0/979435203 conn(0x7f58c406b490 msgr2=0x7f58c406b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c930c640 1 -- 172.21.15.7:0/979435203 shutdown_connections 2024-03-19T01:24:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.371+0000 7f58c930c640 1 -- 172.21.15.7:0/979435203 wait complete. 2024-03-19T01:24:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 Processor -- start 2024-03-19T01:24:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 -- start start 2024-03-19T01:24:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58c406e0e0 0x7f58c41e3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b7a90 0x7f58c4135470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b88c0 0x7f58c41359b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f58c4154710 con 0x7f58c40b88c0 2024-03-19T01:24:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f58c4154590 con 0x7f58c406e0e0 2024-03-19T01:24:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f58c4154890 con 0x7f58c40b7a90 2024-03-19T01:24:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b88c0 0x7f58c41359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b88c0 0x7f58c41359b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48748/0 (socket says 172.21.15.7:48748) 2024-03-19T01:24:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c37fe640 1 -- 172.21.15.7:0/2859427901 learned_addr learned my addr 172.21.15.7:0/2859427901 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58bbfff640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b7a90 0x7f58c4135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c2ffd640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58c406e0e0 0x7f58c41e3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c37fe640 1 -- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b7a90 msgr2=0x7f58c4135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c37fe640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b7a90 0x7f58c4135470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c37fe640 1 -- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58c406e0e0 msgr2=0x7f58c41e3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c37fe640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58c406e0e0 0x7f58c41e3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c37fe640 1 -- 172.21.15.7:0/2859427901 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c40b6850 con 0x7f58c40b88c0 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58bbfff640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b7a90 0x7f58c4135470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c2ffd640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58c406e0e0 0x7f58c41e3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c37fe640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b88c0 0x7f58c41359b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c760 tx=0x7f58bc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58bb7fe640 1 -- 172.21.15.7:0/2859427901 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc003b80 con 0x7f58c40b88c0 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58bb7fe640 1 -- 172.21.15.7:0/2859427901 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58bc003d20 con 0x7f58c40b88c0 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c4145b20 con 0x7f58c40b88c0 2024-03-19T01:24:05.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58bb7fe640 1 -- 172.21.15.7:0/2859427901 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc00aa30 con 0x7f58c40b88c0 2024-03-19T01:24:05.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c41e4710 con 0x7f58c40b88c0 2024-03-19T01:24:05.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c4074130 con 0x7f58c40b88c0 2024-03-19T01:24:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58bb7fe640 1 -- 172.21.15.7:0/2859427901 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f58bc00abd0 con 0x7f58c40b88c0 2024-03-19T01:24:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58bb7fe640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58a4067690 0x7f58a4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58bb7fe640 1 -- 172.21.15.7:0/2859427901 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f58bc0853f0 con 0x7f58c40b88c0 2024-03-19T01:24:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.375+0000 7f58c2ffd640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58a4067690 0x7f58a4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.379+0000 7f58c2ffd640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58a4067690 0x7f58a4069b50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0af000 tx=0x7f58ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:05.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.383+0000 7f58bb7fe640 1 -- 172.21.15.7:0/2859427901 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f58c40b6850 con 0x7f58c40b88c0 2024-03-19T01:24:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.595+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f58c406dcb0 con 0x7f58c40b88c0 2024-03-19T01:24:05.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58bb7fe640 1 -- 172.21.15.7:0/2859427901 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f58bc0563b0 con 0x7f58c40b88c0 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58a4067690 msgr2=0x7f58a4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58a4067690 0x7f58a4069b50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0af000 tx=0x7f58ac008040 comp rx=0 tx=0).stop 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b88c0 msgr2=0x7f58c41359b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b88c0 0x7f58c41359b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c760 tx=0x7f58bc00cc20 comp rx=0 tx=0).stop 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 shutdown_connections 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f58a4067690 0x7f58a4069b50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58c40b7a90 0x7f58c4135470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58c40b88c0 0x7f58c41359b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 --2- 172.21.15.7:0/2859427901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58c406e0e0 0x7f58c41e3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 >> 172.21.15.7:0/2859427901 conn(0x7f58c406b490 msgr2=0x7f58c4133c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 shutdown_connections 2024-03-19T01:24:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.599+0000 7f58c930c640 1 -- 172.21.15.7:0/2859427901 wait complete. 2024-03-19T01:24:05.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-19T01:24:05.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4cb958640 1 Processor -- start 2024-03-19T01:24:05.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4cb958640 1 -- start start 2024-03-19T01:24:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4cb958640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fe4c40b4530 0x7fe4c414a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4cb958640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4c41415b0 0x7fe4c4151d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4cb958640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c40b4910 con 0x7fe4c414aee0 2024-03-19T01:24:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4cb958640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c40b5700 con 0x7fe4c41415b0 2024-03-19T01:24:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4cb958640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c40b6c30 con 0x7fe4c40b4530 2024-03-19T01:24:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4c9ef5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4c41415b0 0x7fe4c4151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4c9ef5640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4c41415b0 0x7fe4c4151d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48758/0 (socket says 172.21.15.7:48758) 2024-03-19T01:24:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4c9ef5640 1 -- 172.21.15.7:0/1913218466 learned_addr learned my addr 172.21.15.7:0/1913218466 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4c96f4640 1 --2- 172.21.15.7:0/1913218466 >> v2:172.21.15.167:3300/0 conn(0x7fe4c40b4530 0x7fe4c414a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4c9ef5640 1 -- 172.21.15.7:0/1913218466 >> v2:172.21.15.167:3300/0 conn(0x7fe4c40b4530 msgr2=0x7fe4c414a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:05.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4c9ef5640 1 --2- 172.21.15.7:0/1913218466 >> v2:172.21.15.167:3300/0 conn(0x7fe4c40b4530 0x7fe4c414a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4c9ef5640 1 -- 172.21.15.7:0/1913218466 >> v1:172.21.15.167:6789/0 conn(0x7fe4c414aee0 legacy=0x7fe4c4140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:05.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.739+0000 7fe4c9ef5640 1 -- 172.21.15.7:0/1913218466 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c40b5b90 con 0x7fe4c41415b0 2024-03-19T01:24:05.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 --2- 172.21.15.7:0/1913218466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c41415b0 0x7fe4c4151d30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40b5880 tx=0x7fe4bc01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eaa7a3979d41869 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:05.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4bbfff640 1 -- 172.21.15.7:0/1913218466 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0a56d0 con 0x7fe4c41415b0 2024-03-19T01:24:05.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4bbfff640 1 -- 172.21.15.7:0/1913218466 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc0a5870 con 0x7fe4c41415b0 2024-03-19T01:24:05.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4bbfff640 1 -- 172.21.15.7:0/1913218466 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0a5c40 con 0x7fe4c41415b0 2024-03-19T01:24:05.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- 172.21.15.7:0/1913218466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c41415b0 msgr2=0x7fe4c4151d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 --2- 172.21.15.7:0/1913218466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c41415b0 0x7fe4c4151d30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40b5880 tx=0x7fe4bc01c900 comp rx=0 tx=0).stop 2024-03-19T01:24:05.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- 172.21.15.7:0/1913218466 shutdown_connections 2024-03-19T01:24:05.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 --2- 172.21.15.7:0/1913218466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c41415b0 0x7fe4c4151d30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 --2- 172.21.15.7:0/1913218466 >> v2:172.21.15.167:3300/0 conn(0x7fe4c40b4530 0x7fe4c414a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- 172.21.15.7:0/1913218466 >> 172.21.15.7:0/1913218466 conn(0x7fe4c4072df0 msgr2=0x7fe4c4073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:05.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- 172.21.15.7:0/1913218466 shutdown_connections 2024-03-19T01:24:05.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- 172.21.15.7:0/1913218466 wait complete. 2024-03-19T01:24:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 Processor -- start 2024-03-19T01:24:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- start start 2024-03-19T01:24:05.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4c40b4530 0x7fe4c413f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4c41415b0 0x7fe4c413b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c414aee0 0x7fe4c413b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c40ba240 con 0x7fe4c414aee0 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c40ba0c0 con 0x7fe4c41415b0 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c4154a90 con 0x7fe4c40b4530 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c414aee0 0x7fe4c413b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c414aee0 0x7fe4c413b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48764/0 (socket says 172.21.15.7:48764) 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 -- 172.21.15.7:0/353478521 learned_addr learned my addr 172.21.15.7:0/353478521 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c8ef3640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4c41415b0 0x7fe4c413b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 -- 172.21.15.7:0/353478521 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4c40b4530 msgr2=0x7fe4c413f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4c40b4530 0x7fe4c413f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 -- 172.21.15.7:0/353478521 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4c41415b0 msgr2=0x7fe4c413b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4c41415b0 0x7fe4c413b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 -- 172.21.15.7:0/353478521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c4142790 con 0x7fe4c414aee0 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4c9ef5640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c414aee0 0x7fe4c413b6e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc01a530 tx=0x7fe4bc0196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4b9ffb640 1 -- 172.21.15.7:0/353478521 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0bc020 con 0x7fe4c414aee0 2024-03-19T01:24:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4b9ffb640 1 -- 172.21.15.7:0/353478521 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc0b8690 con 0x7fe4c414aee0 2024-03-19T01:24:05.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c413e550 con 0x7fe4c414aee0 2024-03-19T01:24:05.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4b9ffb640 1 -- 172.21.15.7:0/353478521 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0b8a80 con 0x7fe4c414aee0 2024-03-19T01:24:05.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.743+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c413d760 con 0x7fe4c414aee0 2024-03-19T01:24:05.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.747+0000 7fe4b9ffb640 1 -- 172.21.15.7:0/353478521 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe4bc0bf020 con 0x7fe4c414aee0 2024-03-19T01:24:05.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.747+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c4075460 con 0x7fe4c414aee0 2024-03-19T01:24:05.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.747+0000 7fe4b9ffb640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe4a8067660 0x7fe4a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:05.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.747+0000 7fe4b9ffb640 1 -- 172.21.15.7:0/353478521 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7fe4bc120550 con 0x7fe4c414aee0 2024-03-19T01:24:05.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.747+0000 7fe4c96f4640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe4a8067660 0x7fe4a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:05.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.747+0000 7fe4c96f4640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe4a8067660 0x7fe4a8069b20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40098f0 tx=0x7fe4b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:05.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.751+0000 7fe4b9ffb640 1 -- 172.21.15.7:0/353478521 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe4bc0c6030 con 0x7fe4c414aee0 2024-03-19T01:24:05.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.979+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fe4c4140910 con 0x7fe4c414aee0 2024-03-19T01:24:05.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4b9ffb640 1 -- 172.21.15.7:0/353478521 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fe4bc016580 con 0x7fe4c414aee0 2024-03-19T01:24:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe4a8067660 msgr2=0x7fe4a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe4a8067660 0x7fe4a8069b20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40098f0 tx=0x7fe4b4008040 comp rx=0 tx=0).stop 2024-03-19T01:24:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c414aee0 msgr2=0x7fe4c413b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c414aee0 0x7fe4c413b6e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc01a530 tx=0x7fe4bc0196a0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 shutdown_connections 2024-03-19T01:24:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe4a8067660 0x7fe4a8069b20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c414aee0 0x7fe4c413b6e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4c41415b0 0x7fe4c413b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 --2- 172.21.15.7:0/353478521 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4c40b4530 0x7fe4c413f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 >> 172.21.15.7:0/353478521 conn(0x7fe4c4072df0 msgr2=0x7fe4c4147420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 shutdown_connections 2024-03-19T01:24:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:05.983+0000 7fe4cb958640 1 -- 172.21.15.7:0/353478521 wait complete. 2024-03-19T01:24:06.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-19T01:24:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c0def6640 1 Processor -- start 2024-03-19T01:24:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c0def6640 1 -- start start 2024-03-19T01:24:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c0def6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5c080ae820 0x7f5c080af050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c0def6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c081495f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c0def6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c08149f80 con 0x7f5c08149b30 2024-03-19T01:24:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c0def6640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f5c080afa70 con 0x7f5c080ae820 2024-03-19T01:24:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c0def6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5c080aec00 con 0x7f5c080af610 2024-03-19T01:24:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c077fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5c080ae820 0x7f5c080af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c077fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5c080ae820 0x7f5c080af050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48770/0 (socket says 172.21.15.7:48770) 2024-03-19T01:24:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c077fe640 1 -- 172.21.15.7:0/1787135136 learned_addr learned my addr 172.21.15.7:0/1787135136 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.123+0000 7f5c06ffd640 1 --2- 172.21.15.7:0/1787135136 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c081495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c077fe640 1 -- 172.21.15.7:0/1787135136 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 msgr2=0x7f5c081495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c077fe640 1 --2- 172.21.15.7:0/1787135136 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c081495f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c077fe640 1 -- 172.21.15.7:0/1787135136 >> v1:172.21.15.167:6789/0 conn(0x7f5c08149b30 legacy=0x7f5c0814ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c077fe640 1 -- 172.21.15.7:0/1787135136 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c080afdd0 con 0x7f5c080ae820 2024-03-19T01:24:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c06ffd640 1 --2- 172.21.15.7:0/1787135136 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c081495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c077fe640 1 --2- 172.21.15.7:0/1787135136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c080ae820 0x7f5c080af050 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009870 tx=0x7f5bf801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473cad1fa4227946 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c067fc640 1 -- 172.21.15.7:0/1787135136 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf80b0070 con 0x7f5c080ae820 2024-03-19T01:24:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c067fc640 1 -- 172.21.15.7:0/1787135136 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bf80ab600 con 0x7f5c080ae820 2024-03-19T01:24:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c067fc640 1 -- 172.21.15.7:0/1787135136 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf80ab9d0 con 0x7f5c080ae820 2024-03-19T01:24:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c0def6640 1 -- 172.21.15.7:0/1787135136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c080ae820 msgr2=0x7f5c080af050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c0def6640 1 --2- 172.21.15.7:0/1787135136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c080ae820 0x7f5c080af050 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009870 tx=0x7f5bf801c440 comp rx=0 tx=0).stop 2024-03-19T01:24:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c0def6640 1 -- 172.21.15.7:0/1787135136 shutdown_connections 2024-03-19T01:24:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c0def6640 1 --2- 172.21.15.7:0/1787135136 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c081495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c0def6640 1 --2- 172.21.15.7:0/1787135136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c080ae820 0x7f5c080af050 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c0def6640 1 -- 172.21.15.7:0/1787135136 >> 172.21.15.7:0/1787135136 conn(0x7f5c080b5330 msgr2=0x7f5c080b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c0def6640 1 -- 172.21.15.7:0/1787135136 shutdown_connections 2024-03-19T01:24:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.127+0000 7f5c0def6640 1 -- 172.21.15.7:0/1787135136 wait complete. 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 Processor -- start 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 -- start start 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c080ae820 0x7f5c08146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c080718f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c08149b30 0x7f5c08071e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5c081544b0 con 0x7f5c08149b30 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5c08154330 con 0x7f5c080ae820 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5c08154630 con 0x7f5c080af610 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c07fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c08149b30 0x7f5c08071e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c07fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c08149b30 0x7f5c08071e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48778/0 (socket says 172.21.15.7:48778) 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c07fff640 1 -- 172.21.15.7:0/1952631060 learned_addr learned my addr 172.21.15.7:0/1952631060 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c06ffd640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c080718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c077fe640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c080ae820 0x7f5c08146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c07fff640 1 -- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 msgr2=0x7f5c080718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c07fff640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c080718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c07fff640 1 -- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c080ae820 msgr2=0x7f5c08146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c07fff640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c080ae820 0x7f5c08146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c07fff640 1 -- 172.21.15.7:0/1952631060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c08147ff0 con 0x7f5c08149b30 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c077fe640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c080ae820 0x7f5c08146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c06ffd640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c080718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c07fff640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c08149b30 0x7f5c08071e30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000c730 tx=0x7f5c0000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5beffff640 1 -- 172.21.15.7:0/1952631060 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00003b80 con 0x7f5c08149b30 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5beffff640 1 -- 172.21.15.7:0/1952631060 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c00003d20 con 0x7f5c08149b30 2024-03-19T01:24:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c08143fe0 con 0x7f5c08149b30 2024-03-19T01:24:06.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5beffff640 1 -- 172.21.15.7:0/1952631060 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000aa30 con 0x7f5c08149b30 2024-03-19T01:24:06.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c081e05c0 con 0x7f5c08149b30 2024-03-19T01:24:06.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c08145500 con 0x7f5c08149b30 2024-03-19T01:24:06.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5beffff640 1 -- 172.21.15.7:0/1952631060 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5c0000abd0 con 0x7f5c08149b30 2024-03-19T01:24:06.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5beffff640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5bf4067690 0x7f5bf4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5beffff640 1 -- 172.21.15.7:0/1952631060 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f5c00085360 con 0x7f5c08149b30 2024-03-19T01:24:06.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.131+0000 7f5c077fe640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5bf4067690 0x7f5bf4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.135+0000 7f5c077fe640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5bf4067690 0x7f5bf4069b50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5bf80ad000 tx=0x7f5bf8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.139+0000 7f5beffff640 1 -- 172.21.15.7:0/1952631060 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5c08145500 con 0x7f5c08149b30 2024-03-19T01:24:06.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.355+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f5c08075460 con 0x7f5c08149b30 2024-03-19T01:24:06.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.359+0000 7f5beffff640 1 -- 172.21.15.7:0/1952631060 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f5c000563c0 con 0x7f5c08149b30 2024-03-19T01:24:06.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5bf4067690 msgr2=0x7f5bf4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5bf4067690 0x7f5bf4069b50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5bf80ad000 tx=0x7f5bf8008040 comp rx=0 tx=0).stop 2024-03-19T01:24:06.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c08149b30 msgr2=0x7f5c08071e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c08149b30 0x7f5c08071e30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000c730 tx=0x7f5c0000cbf0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 shutdown_connections 2024-03-19T01:24:06.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5bf4067690 0x7f5bf4069b50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c080af610 0x7f5c080718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c08149b30 0x7f5c08071e30 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 --2- 172.21.15.7:0/1952631060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c080ae820 0x7f5c08146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 >> 172.21.15.7:0/1952631060 conn(0x7f5c080b5330 msgr2=0x7f5c08072480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 shutdown_connections 2024-03-19T01:24:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.363+0000 7f5c0def6640 1 -- 172.21.15.7:0/1952631060 wait complete. 2024-03-19T01:24:06.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-19T01:24:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa83087e640 1 Processor -- start 2024-03-19T01:24:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa83087e640 1 -- start start 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa83087e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fa82c0b9cb0 0x7fa82c147700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa83087e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa82c0b5f10 0x7fa82c13deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa83087e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82c144170 con 0x7fa82c13e3f0 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa83087e640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fa82c144a40 con 0x7fa82c0b9cb0 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa83087e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa82c070450 con 0x7fa82c0b5f10 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa82ad9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa82c13e3f0 0x7fa82c0709d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51128/0 (socket says 172.21.15.7:51128) 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa82ad9d640 1 -- 172.21.15.7:0/273554775 learned_addr learned my addr 172.21.15.7:0/273554775 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa823fff640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa82c0b5f10 0x7fa82c13deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa82a59c640 1 --2- 172.21.15.7:0/273554775 >> v2:172.21.15.167:3300/0 conn(0x7fa82c0b9cb0 0x7fa82c147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3672346498 0 0) 0x7fa8240014a0 con 0x7fa82c13e3f0 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa818003530 con 0x7fa82c13e3f0 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4050207086 0 0) 0x7fa818003530 con 0x7fa82c13e3f0 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa82c0b5f10 msgr2=0x7fa82c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa829d9b640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa82c0b5f10 0x7fa82c13deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 >> v2:172.21.15.167:3300/0 conn(0x7fa82c0b9cb0 msgr2=0x7fa82c147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.507+0000 7fa829d9b640 1 --2- 172.21.15.7:0/273554775 >> v2:172.21.15.167:3300/0 conn(0x7fa82c0b9cb0 0x7fa82c147700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c0ba090 con 0x7fa82c13e3f0 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fa824001bd0 con 0x7fa82c13e3f0 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 >> v1:172.21.15.7:6789/0 conn(0x7fa82c13e3f0 legacy=0x7fa82c0709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa82c0b5f10 0x7fa8180039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa818003f40 0x7fa818004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa818004d50 0x7fa8180051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c144a40 con 0x7fa818003f40 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c070450 con 0x7fa818004d50 2024-03-19T01:24:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa81800ba20 con 0x7fa82c0b5f10 2024-03-19T01:24:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa823fff640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa818003f40 0x7fa818004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa82a59c640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa82c0b5f10 0x7fa8180039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa82ad9d640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa818004d50 0x7fa8180051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa823fff640 1 -- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa82c0b5f10 msgr2=0x7fa8180039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa823fff640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa82c0b5f10 0x7fa8180039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa823fff640 1 -- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa818004d50 msgr2=0x7fa8180051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa823fff640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa818004d50 0x7fa8180051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa823fff640 1 -- 172.21.15.7:0/273554775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa818003530 con 0x7fa818003f40 2024-03-19T01:24:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa82a59c640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa82c0b5f10 0x7fa8180039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa823fff640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa818003f40 0x7fa818004810 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa814009b10 tx=0x7fa81401ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45429f151cab08b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140b2070 con 0x7fa818003f40 2024-03-19T01:24:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8140a5520 con 0x7fa818003f40 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.511+0000 7fa829d9b640 1 -- 172.21.15.7:0/273554775 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140096b0 con 0x7fa818003f40 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- 172.21.15.7:0/273554775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa818003f40 msgr2=0x7fa818004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 --2- 172.21.15.7:0/273554775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa818003f40 0x7fa818004810 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa814009b10 tx=0x7fa81401ca40 comp rx=0 tx=0).stop 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa82ad9d640 1 -- 172.21.15.7:0/273554775 reap_dead start 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- 172.21.15.7:0/273554775 shutdown_connections 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- 172.21.15.7:0/273554775 >> 172.21.15.7:0/273554775 conn(0x7fa82c074130 msgr2=0x7fa82c074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- 172.21.15.7:0/273554775 shutdown_connections 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- 172.21.15.7:0/273554775 wait complete. 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 Processor -- start 2024-03-19T01:24:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- start start 2024-03-19T01:24:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa818003f40 0x7fa82c1d4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa82c0b16e0 0x7fa82c1e11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa82c1e1710 0x7fa82c13fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c13e7d0 con 0x7fa82c0b16e0 2024-03-19T01:24:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c147c40 con 0x7fa82c1e1710 2024-03-19T01:24:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa82c1e1bc0 con 0x7fa818003f40 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa823fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa82c0b16e0 0x7fa82c1e11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa823fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa82c0b16e0 0x7fa82c1e11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48806/0 (socket says 172.21.15.7:48806) 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa823fff640 1 -- 172.21.15.7:0/2067495038 learned_addr learned my addr 172.21.15.7:0/2067495038 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa82a59c640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa818003f40 0x7fa82c1d4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa823fff640 1 -- 172.21.15.7:0/2067495038 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa818003f40 msgr2=0x7fa82c1d4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa823fff640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa818003f40 0x7fa82c1d4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa82ad9d640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa82c1e1710 0x7fa82c13fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa823fff640 1 -- 172.21.15.7:0/2067495038 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa82c1e1710 msgr2=0x7fa82c13fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa823fff640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa82c1e1710 0x7fa82c13fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa823fff640 1 -- 172.21.15.7:0/2067495038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c0b1bc0 con 0x7fa82c0b16e0 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa823fff640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa82c0b16e0 0x7fa82c1e11d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa81401c870 tx=0x7fa818005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa822ffd640 1 -- 172.21.15.7:0/2067495038 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140bc030 con 0x7fa82c0b16e0 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa822ffd640 1 -- 172.21.15.7:0/2067495038 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa81400f000 con 0x7fa82c0b16e0 2024-03-19T01:24:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82c0b1ea0 con 0x7fa82c0b16e0 2024-03-19T01:24:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa822ffd640 1 -- 172.21.15.7:0/2067495038 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140b2040 con 0x7fa82c0b16e0 2024-03-19T01:24:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.515+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82c1413b0 con 0x7fa82c0b16e0 2024-03-19T01:24:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.519+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa82c075460 con 0x7fa82c0b16e0 2024-03-19T01:24:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.519+0000 7fa822ffd640 1 -- 172.21.15.7:0/2067495038 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa8140bf070 con 0x7fa82c0b16e0 2024-03-19T01:24:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.519+0000 7fa822ffd640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa80c067660 0x7fa80c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.519+0000 7fa822ffd640 1 -- 172.21.15.7:0/2067495038 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7fa81411f060 con 0x7fa82c0b16e0 2024-03-19T01:24:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.519+0000 7fa82a59c640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa80c067660 0x7fa80c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.519+0000 7fa82a59c640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa80c067660 0x7fa80c069b20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009b30 tx=0x7fa81c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:06.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.523+0000 7fa822ffd640 1 -- 172.21.15.7:0/2067495038 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa8140c7020 con 0x7fa82c0b16e0 2024-03-19T01:24:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.739+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fa82c0b91f0 con 0x7fa82c0b16e0 2024-03-19T01:24:06.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.743+0000 7fa822ffd640 1 -- 172.21.15.7:0/2067495038 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fa8140167c0 con 0x7fa82c0b16e0 2024-03-19T01:24:06.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa80c067660 msgr2=0x7fa80c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa80c067660 0x7fa80c069b20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009b30 tx=0x7fa81c008040 comp rx=0 tx=0).stop 2024-03-19T01:24:06.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa82c0b16e0 msgr2=0x7fa82c1e11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa82c0b16e0 0x7fa82c1e11d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa81401c870 tx=0x7fa818005840 comp rx=0 tx=0).stop 2024-03-19T01:24:06.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 shutdown_connections 2024-03-19T01:24:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa80c067660 0x7fa80c069b20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa82c1e1710 0x7fa82c13fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa82c0b16e0 0x7fa82c1e11d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 --2- 172.21.15.7:0/2067495038 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa818003f40 0x7fa82c1d4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 >> 172.21.15.7:0/2067495038 conn(0x7fa82c074130 msgr2=0x7fa82c0b6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 shutdown_connections 2024-03-19T01:24:06.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.747+0000 7fa83087e640 1 -- 172.21.15.7:0/2067495038 wait complete. 2024-03-19T01:24:06.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-19T01:24:06.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9c447640 1 Processor -- start 2024-03-19T01:24:06.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9c447640 1 -- start start 2024-03-19T01:24:06.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9c447640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b9406f1c0 0x7f4b9413f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9c447640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f640 0x7f4b9414fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9c447640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b940b85a0 con 0x7f4b940b81c0 2024-03-19T01:24:06.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9c447640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f4b940b2710 con 0x7f4b9406f1c0 2024-03-19T01:24:06.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9c447640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4b94070960 con 0x7f4b9406f640 2024-03-19T01:24:06.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b999e2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b9406f1c0 0x7f4b9413f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9a1e3640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4b940b81c0 0x7f4b941486e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51132/0 (socket says 172.21.15.7:51132) 2024-03-19T01:24:06.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9a1e3640 1 -- 172.21.15.7:0/2475990932 learned_addr learned my addr 172.21.15.7:0/2475990932 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:06.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b9a9e4640 1 --2- 172.21.15.7:0/2475990932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f640 0x7f4b9414fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b999e2640 1 -- 172.21.15.7:0/2475990932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f640 msgr2=0x7f4b9414fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:06.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b999e2640 1 --2- 172.21.15.7:0/2475990932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f640 0x7f4b9414fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b999e2640 1 -- 172.21.15.7:0/2475990932 >> v1:172.21.15.7:6789/0 conn(0x7f4b940b81c0 legacy=0x7f4b941486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b999e2640 1 -- 172.21.15.7:0/2475990932 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b940b20c0 con 0x7f4b9406f1c0 2024-03-19T01:24:06.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.887+0000 7f4b991e1640 1 -- 172.21.15.7:0/2475990932 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3091345814 0 0) 0x7f4b940b85a0 con 0x7f4b940b81c0 2024-03-19T01:24:06.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 --2- 172.21.15.7:0/2475990932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f1c0 0x7f4b9413f0a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4b840097c0 tx=0x7f4b8401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e823b79763eccf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:06.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b991e1640 1 -- 172.21.15.7:0/2475990932 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840b0070 con 0x7f4b9406f1c0 2024-03-19T01:24:06.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b991e1640 1 -- 172.21.15.7:0/2475990932 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b840ab600 con 0x7f4b9406f1c0 2024-03-19T01:24:06.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b991e1640 1 -- 172.21.15.7:0/2475990932 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840ab9d0 con 0x7f4b9406f1c0 2024-03-19T01:24:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- 172.21.15.7:0/2475990932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f1c0 msgr2=0x7f4b9413f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 --2- 172.21.15.7:0/2475990932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f1c0 0x7f4b9413f0a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4b840097c0 tx=0x7f4b8401c410 comp rx=0 tx=0).stop 2024-03-19T01:24:06.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- 172.21.15.7:0/2475990932 shutdown_connections 2024-03-19T01:24:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 --2- 172.21.15.7:0/2475990932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f640 0x7f4b9414fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 --2- 172.21.15.7:0/2475990932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f1c0 0x7f4b9413f0a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- 172.21.15.7:0/2475990932 >> 172.21.15.7:0/2475990932 conn(0x7f4b940b5330 msgr2=0x7f4b940b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- 172.21.15.7:0/2475990932 shutdown_connections 2024-03-19T01:24:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- 172.21.15.7:0/2475990932 wait complete. 2024-03-19T01:24:06.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 Processor -- start 2024-03-19T01:24:06.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- start start 2024-03-19T01:24:06.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f1c0 0x7f4b940b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f640 0x7f4b941e11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b940b81c0 0x7f4b9412f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4b94152b60 con 0x7f4b9406f640 2024-03-19T01:24:06.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4b941529e0 con 0x7f4b940b81c0 2024-03-19T01:24:06.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4b94152ce0 con 0x7f4b9406f1c0 2024-03-19T01:24:06.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f640 0x7f4b941e11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f640 0x7f4b941e11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48822/0 (socket says 172.21.15.7:48822) 2024-03-19T01:24:06.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 -- 172.21.15.7:0/2663912267 learned_addr learned my addr 172.21.15.7:0/2663912267 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:06.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9a1e3640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f1c0 0x7f4b940b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9a9e4640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b940b81c0 0x7f4b9412f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 -- 172.21.15.7:0/2663912267 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f1c0 msgr2=0x7f4b940b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f1c0 0x7f4b940b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 -- 172.21.15.7:0/2663912267 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b940b81c0 msgr2=0x7f4b9412f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b940b81c0 0x7f4b9412f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 -- 172.21.15.7:0/2663912267 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b940b4ae0 con 0x7f4b9406f640 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b999e2640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f640 0x7f4b941e11b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4b840097c0 tx=0x7f4b8401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b8affd640 1 -- 172.21.15.7:0/2663912267 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840bc070 con 0x7f4b9406f640 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b8affd640 1 -- 172.21.15.7:0/2663912267 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b840b1040 con 0x7f4b9406f640 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b8affd640 1 -- 172.21.15.7:0/2663912267 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840b0050 con 0x7f4b9406f640 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b940b4000 con 0x7f4b9406f640 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.891+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b94137740 con 0x7f4b9406f640 2024-03-19T01:24:06.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.895+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b94075460 con 0x7f4b9406f640 2024-03-19T01:24:06.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.895+0000 7f4b8affd640 1 -- 172.21.15.7:0/2663912267 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4b840bf070 con 0x7f4b9406f640 2024-03-19T01:24:06.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.895+0000 7f4b8affd640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b78067690 0x7f4b78069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:06.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.895+0000 7f4b8affd640 1 -- 172.21.15.7:0/2663912267 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f4b8411f120 con 0x7f4b9406f640 2024-03-19T01:24:06.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.895+0000 7f4b9a1e3640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b78067690 0x7f4b78069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:06.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.895+0000 7f4b9a1e3640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b78067690 0x7f4b78069b50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4b90000f70 tx=0x7f4b90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:06.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:06.899+0000 7f4b8affd640 1 -- 172.21.15.7:0/2663912267 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4b84016090 con 0x7f4b9406f640 2024-03-19T01:24:07.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.123+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f4b9413ac00 con 0x7f4b9406f640 2024-03-19T01:24:07.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b8affd640 1 -- 172.21.15.7:0/2663912267 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f4b840f00e0 con 0x7f4b9406f640 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b78067690 msgr2=0x7f4b78069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b78067690 0x7f4b78069b50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4b90000f70 tx=0x7f4b90009040 comp rx=0 tx=0).stop 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f640 msgr2=0x7f4b941e11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f640 0x7f4b941e11b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4b840097c0 tx=0x7f4b8401ac50 comp rx=0 tx=0).stop 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 shutdown_connections 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4b78067690 0x7f4b78069b50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b940b81c0 0x7f4b9412f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b9406f640 0x7f4b941e11b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 --2- 172.21.15.7:0/2663912267 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b9406f1c0 0x7f4b940b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 >> 172.21.15.7:0/2663912267 conn(0x7f4b940b5330 msgr2=0x7f4b94139890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 shutdown_connections 2024-03-19T01:24:07.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.127+0000 7f4b9c447640 1 -- 172.21.15.7:0/2663912267 wait complete. 2024-03-19T01:24:07.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-19T01:24:07.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f124e44b640 1 Processor -- start 2024-03-19T01:24:07.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f124e44b640 1 -- start start 2024-03-19T01:24:07.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f124e44b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f124813dd70 0x7f1248143830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f124e44b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12480b2ca0 con 0x7f12480b1310 2024-03-19T01:24:07.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f124e44b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12480b7080 con 0x7f12480b16f0 2024-03-19T01:24:07.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f124e44b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f12480b29c0 con 0x7f124813dd70 2024-03-19T01:24:07.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f1247fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f12480b1310 0x7f124813d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51142/0 (socket says 172.21.15.7:51142) 2024-03-19T01:24:07.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f1247fff640 1 -- 172.21.15.7:0/3607847921 learned_addr learned my addr 172.21.15.7:0/3607847921 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:07.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f12477fe640 1 --2- 172.21.15.7:0/3607847921 >> v2:172.21.15.7:3300/0 conn(0x7f124813dd70 0x7f1248143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f1246ffd640 1 -- 172.21.15.7:0/3607847921 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 656717906 0 0) 0x7f12480b2ca0 con 0x7f12480b1310 2024-03-19T01:24:07.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f1246ffd640 1 -- 172.21.15.7:0/3607847921 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1230003530 con 0x7f12480b1310 2024-03-19T01:24:07.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f12477fe640 1 -- 172.21.15.7:0/3607847921 >> v1:172.21.15.167:6789/0 conn(0x7f12480b16f0 legacy=0x7f124814e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:07.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f12477fe640 1 -- 172.21.15.7:0/3607847921 >> v1:172.21.15.7:6789/0 conn(0x7f12480b1310 legacy=0x7f124813d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.259+0000 7f12477fe640 1 -- 172.21.15.7:0/3607847921 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12480b7320 con 0x7f124813dd70 2024-03-19T01:24:07.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f12477fe640 1 --2- 172.21.15.7:0/3607847921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f124813dd70 0x7f1248143830 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f12380097c0 tx=0x7f123801c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210b39a64bbf412d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:07.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f1246ffd640 1 -- 172.21.15.7:0/3607847921 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12380b0070 con 0x7f124813dd70 2024-03-19T01:24:07.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f1246ffd640 1 -- 172.21.15.7:0/3607847921 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12380ab600 con 0x7f124813dd70 2024-03-19T01:24:07.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f1246ffd640 1 -- 172.21.15.7:0/3607847921 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12380ab9b0 con 0x7f124813dd70 2024-03-19T01:24:07.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f124e44b640 1 -- 172.21.15.7:0/3607847921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f124813dd70 msgr2=0x7f1248143830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f124e44b640 1 --2- 172.21.15.7:0/3607847921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f124813dd70 0x7f1248143830 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f12380097c0 tx=0x7f123801c350 comp rx=0 tx=0).stop 2024-03-19T01:24:07.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f124e44b640 1 -- 172.21.15.7:0/3607847921 shutdown_connections 2024-03-19T01:24:07.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f124e44b640 1 --2- 172.21.15.7:0/3607847921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f124813dd70 0x7f1248143830 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f124e44b640 1 -- 172.21.15.7:0/3607847921 >> 172.21.15.7:0/3607847921 conn(0x7f12480b5330 msgr2=0x7f12480b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:07.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f124e44b640 1 -- 172.21.15.7:0/3607847921 shutdown_connections 2024-03-19T01:24:07.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f124e44b640 1 -- 172.21.15.7:0/3607847921 wait complete. 2024-03-19T01:24:07.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f124e44b640 1 Processor -- start 2024-03-19T01:24:07.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.263+0000 7f124e44b640 1 -- start start 2024-03-19T01:24:07.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f124e44b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12480b1310 0x7f1248135820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f124e44b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12480b16f0 0x7f1248071770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f124e44b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f124813dd70 0x7f12481df430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f124e44b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f12481519d0 con 0x7f12480b16f0 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f124e44b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1248151850 con 0x7f12480b1310 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f124e44b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1248151b50 con 0x7f124813dd70 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f12477fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12480b16f0 0x7f1248071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f12477fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12480b16f0 0x7f1248071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48840/0 (socket says 172.21.15.7:48840) 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f12477fe640 1 -- 172.21.15.7:0/1097514089 learned_addr learned my addr 172.21.15.7:0/1097514089 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f12477fe640 1 -- 172.21.15.7:0/1097514089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f124813dd70 msgr2=0x7f12481df430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f12477fe640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f124813dd70 0x7f12481df430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f12477fe640 1 -- 172.21.15.7:0/1097514089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12480b1310 msgr2=0x7f1248135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f12477fe640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12480b1310 0x7f1248135820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f12477fe640 1 -- 172.21.15.7:0/1097514089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1248130f90 con 0x7f12480b16f0 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f12477fe640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12480b16f0 0x7f1248071770 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f12380097c0 tx=0x7f123801ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f1244ff9640 1 -- 172.21.15.7:0/1097514089 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12380b0070 con 0x7f12480b16f0 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f1244ff9640 1 -- 172.21.15.7:0/1097514089 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12380ba070 con 0x7f12480b16f0 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f1244ff9640 1 -- 172.21.15.7:0/1097514089 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12380abbc0 con 0x7f12480b16f0 2024-03-19T01:24:07.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12481322f0 con 0x7f12480b16f0 2024-03-19T01:24:07.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1248131240 con 0x7f12480b16f0 2024-03-19T01:24:07.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1248075460 con 0x7f12480b16f0 2024-03-19T01:24:07.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f1244ff9640 1 -- 172.21.15.7:0/1097514089 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1238003620 con 0x7f12480b16f0 2024-03-19T01:24:07.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f1244ff9640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1234067660 0x7f1234069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f1244ff9640 1 -- 172.21.15.7:0/1097514089 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f123811f490 con 0x7f12480b16f0 2024-03-19T01:24:07.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f1247fff640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1234067660 0x7f1234069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.267+0000 7f1247fff640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1234067660 0x7f1234069b20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f123c007fd0 tx=0x7f123c007940 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:07.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.271+0000 7f1244ff9640 1 -- 172.21.15.7:0/1097514089 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1238016090 con 0x7f12480b16f0 2024-03-19T01:24:07.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.491+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f1248141480 con 0x7f12480b16f0 2024-03-19T01:24:07.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.491+0000 7f1244ff9640 1 -- 172.21.15.7:0/1097514089 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f12380f04f0 con 0x7f12480b16f0 2024-03-19T01:24:07.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1234067660 msgr2=0x7f1234069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1234067660 0x7f1234069b20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f123c007fd0 tx=0x7f123c007940 comp rx=0 tx=0).stop 2024-03-19T01:24:07.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12480b16f0 msgr2=0x7f1248071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12480b16f0 0x7f1248071770 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f12380097c0 tx=0x7f123801ab90 comp rx=0 tx=0).stop 2024-03-19T01:24:07.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 shutdown_connections 2024-03-19T01:24:07.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1234067660 0x7f1234069b20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f124813dd70 0x7f12481df430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12480b16f0 0x7f1248071770 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 --2- 172.21.15.7:0/1097514089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12480b1310 0x7f1248135820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 >> 172.21.15.7:0/1097514089 conn(0x7f12480b5330 msgr2=0x7f12480b8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:07.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 shutdown_connections 2024-03-19T01:24:07.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.495+0000 7f124e44b640 1 -- 172.21.15.7:0/1097514089 wait complete. 2024-03-19T01:24:07.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:07.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.631+0000 7f1a34acc640 1 Processor -- start 2024-03-19T01:24:07.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.631+0000 7f1a34acc640 1 -- start start 2024-03-19T01:24:07.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.631+0000 7f1a34acc640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f1a30071780 0x7f1a300b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.631+0000 7f1a34acc640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a301454f0 con 0x7f1a30145cf0 2024-03-19T01:24:07.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.631+0000 7f1a34acc640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a301451e0 con 0x7f1a30071b60 2024-03-19T01:24:07.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.631+0000 7f1a34acc640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f1a30145860 con 0x7f1a30071780 2024-03-19T01:24:07.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.631+0000 7f1a2ed9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1a30145cf0 0x7f1a3013ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51150/0 (socket says 172.21.15.7:51150) 2024-03-19T01:24:07.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.631+0000 7f1a2ed9d640 1 -- 172.21.15.7:0/902435848 learned_addr learned my addr 172.21.15.7:0/902435848 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.631+0000 7f1a2e59c640 1 --2- 172.21.15.7:0/902435848 >> v2:172.21.15.167:3300/0 conn(0x7f1a30071780 0x7f1a300b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3155921355 0 0) 0x7f1a301454f0 con 0x7f1a30145cf0 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a1c003530 con 0x7f1a30145cf0 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 958540373 0 0) 0x7f1a1c003530 con 0x7f1a30145cf0 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 >> v2:172.21.15.167:3300/0 conn(0x7f1a30071780 msgr2=0x7f1a300b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 --2- 172.21.15.7:0/902435848 >> v2:172.21.15.167:3300/0 conn(0x7f1a30071780 0x7f1a300b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 >> v1:172.21.15.167:6789/0 conn(0x7f1a30071b60 legacy=0x7f1a30072da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3013f660 con 0x7f1a30145cf0 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f1a280016a0 con 0x7f1a30145cf0 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 >> v1:172.21.15.7:6789/0 conn(0x7f1a30145cf0 legacy=0x7f1a3013ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a1c0039c0 0x7f1a1c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1c004810 0x7f1a1c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0051d0 0x7f1a1c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1a30145860 con 0x7f1a1c0039c0 2024-03-19T01:24:07.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1c00c170 con 0x7f1a1c0051d0 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1c00c370 con 0x7f1a1c004810 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2e59c640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a1c0039c0 0x7f1a1c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2dd9b640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1c004810 0x7f1a1c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2ed9d640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0051d0 0x7f1a1c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2e59c640 1 -- 172.21.15.7:0/902435848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1c004810 msgr2=0x7f1a1c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2e59c640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1c004810 0x7f1a1c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2e59c640 1 -- 172.21.15.7:0/902435848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0051d0 msgr2=0x7f1a1c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2e59c640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0051d0 0x7f1a1c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2e59c640 1 -- 172.21.15.7:0/902435848 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1c003530 con 0x7f1a1c0039c0 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2ed9d640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0051d0 0x7f1a1c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2ed9d640 1 -- 172.21.15.7:0/902435848 reap_dead start 2024-03-19T01:24:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2e59c640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a1c0039c0 0x7f1a1c003de0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009fc0 tx=0x7f1a2001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547f8516adf85946 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a200b2070 con 0x7f1a1c0039c0 2024-03-19T01:24:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a200a5a70 con 0x7f1a1c0039c0 2024-03-19T01:24:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a2d59a640 1 -- 172.21.15.7:0/902435848 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20009c10 con 0x7f1a1c0039c0 2024-03-19T01:24:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a34acc640 1 -- 172.21.15.7:0/902435848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a1c0039c0 msgr2=0x7f1a1c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a34acc640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a1c0039c0 0x7f1a1c003de0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009fc0 tx=0x7f1a2001cc80 comp rx=0 tx=0).stop 2024-03-19T01:24:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a34acc640 1 -- 172.21.15.7:0/902435848 shutdown_connections 2024-03-19T01:24:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a34acc640 1 --2- 172.21.15.7:0/902435848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a1c0039c0 0x7f1a1c003de0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.635+0000 7f1a34acc640 1 -- 172.21.15.7:0/902435848 >> 172.21.15.7:0/902435848 conn(0x7f1a3006bc10 msgr2=0x7f1a300b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 -- 172.21.15.7:0/902435848 shutdown_connections 2024-03-19T01:24:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 -- 172.21.15.7:0/902435848 wait complete. 2024-03-19T01:24:07.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 Processor -- start 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 -- start start 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0039c0 0x7f1a30146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a3013a990 0x7f1a301de390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a301de8d0 0x7f1a30135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3013ef50 con 0x7f1a3013a990 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a300b2650 con 0x7f1a1c0039c0 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a3013f0d0 con 0x7f1a301de8d0 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a3013a990 0x7f1a301de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a3013a990 0x7f1a301de390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48850/0 (socket says 172.21.15.7:48850) 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2dd9b640 1 -- 172.21.15.7:0/1954986270 learned_addr learned my addr 172.21.15.7:0/1954986270 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2e59c640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0039c0 0x7f1a30146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2ed9d640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a301de8d0 0x7f1a30135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2dd9b640 1 -- 172.21.15.7:0/1954986270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a301de8d0 msgr2=0x7f1a30135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2dd9b640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a301de8d0 0x7f1a30135bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2dd9b640 1 -- 172.21.15.7:0/1954986270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0039c0 msgr2=0x7f1a30146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2dd9b640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0039c0 0x7f1a30146600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2dd9b640 1 -- 172.21.15.7:0/1954986270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a30144a90 con 0x7f1a3013a990 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2ed9d640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a301de8d0 0x7f1a30135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a2dd9b640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a3013a990 0x7f1a301de390 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800e4d0 tx=0x7f1a1800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a26ffd640 1 -- 172.21.15.7:0/1954986270 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a180053a0 con 0x7f1a3013a990 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a26ffd640 1 -- 172.21.15.7:0/1954986270 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a18018070 con 0x7f1a3013a990 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a26ffd640 1 -- 172.21.15.7:0/1954986270 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18003820 con 0x7f1a3013a990 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a30148390 con 0x7f1a3013a990 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a30147a10 con 0x7f1a3013a990 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.639+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a30148180 con 0x7f1a3013a990 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.643+0000 7f1a26ffd640 1 -- 172.21.15.7:0/1954986270 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1a180039c0 con 0x7f1a3013a990 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.643+0000 7f1a26ffd640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a10067660 0x7f1a10069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.643+0000 7f1a2e59c640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a10067660 0x7f1a10069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.643+0000 7f1a26ffd640 1 -- 172.21.15.7:0/1954986270 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f1a18085180 con 0x7f1a3013a990 2024-03-19T01:24:07.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.643+0000 7f1a2e59c640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a10067660 0x7f1a10069b20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1a200af000 tx=0x7f1a20004ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:07.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.647+0000 7f1a26ffd640 1 -- 172.21.15.7:0/1954986270 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1a180561e0 con 0x7f1a3013a990 2024-03-19T01:24:07.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.807+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a30075460 con 0x7f1a3013a990 2024-03-19T01:24:07.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.807+0000 7f1a26ffd640 1 -- 172.21.15.7:0/1954986270 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+12855 (secure 0 0 0) 0x7f1a18083e20 con 0x7f1a3013a990 2024-03-19T01:24:07.810 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:07.810 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":26,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:24:04.838385+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a10067660 msgr2=0x7f1a10069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a10067660 0x7f1a10069b20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1a200af000 tx=0x7f1a20004ac0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a3013a990 msgr2=0x7f1a301de390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a3013a990 0x7f1a301de390 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800e4d0 tx=0x7f1a1800e990 comp rx=0 tx=0).stop 2024-03-19T01:24:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 shutdown_connections 2024-03-19T01:24:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a10067660 0x7f1a10069b20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a301de8d0 0x7f1a30135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a3013a990 0x7f1a301de390 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 --2- 172.21.15.7:0/1954986270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1c0039c0 0x7f1a30146600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 >> 172.21.15.7:0/1954986270 conn(0x7f1a3006bc10 msgr2=0x7f1a300b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:07.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 shutdown_connections 2024-03-19T01:24:07.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.811+0000 7f1a34acc640 1 -- 172.21.15.7:0/1954986270 wait complete. 2024-03-19T01:24:07.829 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:24:07.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:24:07.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f24a1219640 1 Processor -- start 2024-03-19T01:24:07.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f24a1219640 1 -- start start 2024-03-19T01:24:07.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f24a1219640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c0b59a0 0x7f249c1450d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f24a1219640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f249c06ff90 0x7f249c071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f24a1219640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c0b5d80 con 0x7f249c072440 2024-03-19T01:24:07.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f24a1219640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f249c145610 con 0x7f249c06ff90 2024-03-19T01:24:07.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f24a1219640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f249c0704c0 con 0x7f249c0b59a0 2024-03-19T01:24:07.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f2493fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f249c06ff90 0x7f249c071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f249b59e640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f249c072440 0x7f249c0b1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51156/0 (socket says 172.21.15.7:51156) 2024-03-19T01:24:07.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f249b59e640 1 -- 172.21.15.7:0/2433593761 learned_addr learned my addr 172.21.15.7:0/2433593761 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:07.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f249ad9d640 1 --2- 172.21.15.7:0/2433593761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c0b59a0 0x7f249c1450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f249a59c640 1 -- 172.21.15.7:0/2433593761 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1907882720 0 0) 0x7f249c0b5d80 con 0x7f249c072440 2024-03-19T01:24:07.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f249a59c640 1 -- 172.21.15.7:0/2433593761 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2488003530 con 0x7f249c072440 2024-03-19T01:24:07.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f2493fff640 1 -- 172.21.15.7:0/2433593761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c0b59a0 msgr2=0x7f249c1450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:07.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f2493fff640 1 --2- 172.21.15.7:0/2433593761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c0b59a0 0x7f249c1450d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f2493fff640 1 -- 172.21.15.7:0/2433593761 >> v1:172.21.15.7:6789/0 conn(0x7f249c072440 legacy=0x7f249c0b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.959+0000 7f2493fff640 1 -- 172.21.15.7:0/2433593761 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c072a80 con 0x7f249c06ff90 2024-03-19T01:24:07.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f2493fff640 1 --2- 172.21.15.7:0/2433593761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c06ff90 0x7f249c071f00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f24840097c0 tx=0x7f248401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79f488016284522 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:07.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249a59c640 1 -- 172.21.15.7:0/2433593761 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24840b0070 con 0x7f249c06ff90 2024-03-19T01:24:07.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249a59c640 1 -- 172.21.15.7:0/2433593761 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24840ab400 con 0x7f249c06ff90 2024-03-19T01:24:07.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249a59c640 1 -- 172.21.15.7:0/2433593761 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24840ab7d0 con 0x7f249c06ff90 2024-03-19T01:24:07.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 -- 172.21.15.7:0/2433593761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c06ff90 msgr2=0x7f249c071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 --2- 172.21.15.7:0/2433593761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c06ff90 0x7f249c071f00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f24840097c0 tx=0x7f248401c380 comp rx=0 tx=0).stop 2024-03-19T01:24:07.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 -- 172.21.15.7:0/2433593761 shutdown_connections 2024-03-19T01:24:07.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 --2- 172.21.15.7:0/2433593761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c06ff90 0x7f249c071f00 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 --2- 172.21.15.7:0/2433593761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c0b59a0 0x7f249c1450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 -- 172.21.15.7:0/2433593761 >> 172.21.15.7:0/2433593761 conn(0x7f249c06c3b0 msgr2=0x7f249c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:07.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 -- 172.21.15.7:0/2433593761 shutdown_connections 2024-03-19T01:24:07.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 -- 172.21.15.7:0/2433593761 wait complete. 2024-03-19T01:24:07.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 Processor -- start 2024-03-19T01:24:07.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 -- start start 2024-03-19T01:24:07.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f249c06ff90 0x7f249c13a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c072440 0x7f249c13b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c0b59a0 0x7f249c1ddbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f249c0b4f50 con 0x7f249c0b59a0 2024-03-19T01:24:07.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f249c0b4dd0 con 0x7f249c06ff90 2024-03-19T01:24:07.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f24a1219640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f249c152a70 con 0x7f249c072440 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249b59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c0b59a0 0x7f249c1ddbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249b59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c0b59a0 0x7f249c1ddbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48872/0 (socket says 172.21.15.7:48872) 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249b59e640 1 -- 172.21.15.7:0/727128859 learned_addr learned my addr 172.21.15.7:0/727128859 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f2493fff640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c072440 0x7f249c13b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249ad9d640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f249c06ff90 0x7f249c13a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249b59e640 1 -- 172.21.15.7:0/727128859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c072440 msgr2=0x7f249c13b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249b59e640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c072440 0x7f249c13b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249b59e640 1 -- 172.21.15.7:0/727128859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f249c06ff90 msgr2=0x7f249c13a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249b59e640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f249c06ff90 0x7f249c13a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249b59e640 1 -- 172.21.15.7:0/727128859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c13b960 con 0x7f249c0b59a0 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f2493fff640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c072440 0x7f249c13b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.963+0000 7f249b59e640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c0b59a0 0x7f249c1ddbd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f249400cae0 tx=0x7f249400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f24937fe640 1 -- 172.21.15.7:0/727128859 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494017070 con 0x7f249c0b59a0 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f24937fe640 1 -- 172.21.15.7:0/727128859 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249400ac40 con 0x7f249c0b59a0 2024-03-19T01:24:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f249c1396e0 con 0x7f249c0b59a0 2024-03-19T01:24:07.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f24937fe640 1 -- 172.21.15.7:0/727128859 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494012640 con 0x7f249c0b59a0 2024-03-19T01:24:07.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f249c13ea10 con 0x7f249c0b59a0 2024-03-19T01:24:07.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f249c075570 con 0x7f249c0b59a0 2024-03-19T01:24:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f24937fe640 1 -- 172.21.15.7:0/727128859 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f249401a070 con 0x7f249c0b59a0 2024-03-19T01:24:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f24937fe640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f247c067660 0x7f247c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f249ad9d640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f247c067660 0x7f247c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f24937fe640 1 -- 172.21.15.7:0/727128859 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f2494085350 con 0x7f249c0b59a0 2024-03-19T01:24:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.967+0000 7f249ad9d640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f247c067660 0x7f247c069b20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f248c009a50 tx=0x7f248c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:07.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:07.971+0000 7f24937fe640 1 -- 172.21.15.7:0/727128859 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f249c13b960 con 0x7f249c0b59a0 2024-03-19T01:24:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f249c06b8e0 con 0x7f249c0b59a0 2024-03-19T01:24:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24937fe640 1 -- 172.21.15.7:0/727128859 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f2494056310 con 0x7f249c0b59a0 2024-03-19T01:24:08.128 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:24:08.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f247c067660 msgr2=0x7f247c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f247c067660 0x7f247c069b20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f248c009a50 tx=0x7f248c008040 comp rx=0 tx=0).stop 2024-03-19T01:24:08.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c0b59a0 msgr2=0x7f249c1ddbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c0b59a0 0x7f249c1ddbd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f249400cae0 tx=0x7f249400a060 comp rx=0 tx=0).stop 2024-03-19T01:24:08.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 shutdown_connections 2024-03-19T01:24:08.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f247c067660 0x7f247c069b20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f249c072440 0x7f249c13b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c0b59a0 0x7f249c1ddbd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 --2- 172.21.15.7:0/727128859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f249c06ff90 0x7f249c13a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 >> 172.21.15.7:0/727128859 conn(0x7f249c06c3b0 msgr2=0x7f249c0b8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:08.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 shutdown_connections 2024-03-19T01:24:08.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.127+0000 7f24a1219640 1 -- 172.21.15.7:0/727128859 wait complete. 2024-03-19T01:24:08.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-19T01:24:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca9a22640 1 Processor -- start 2024-03-19T01:24:08.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca9a22640 1 -- start start 2024-03-19T01:24:08.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca9a22640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f9ca4074390 0x7f9ca414aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca9a22640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca4144cc0 0x7f9ca40aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca9a22640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca40afed0 con 0x7f9ca40ad870 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca9a22640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f9ca40b0140 con 0x7f9ca4074390 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca9a22640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca4074770 con 0x7f9ca4144cc0 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca27fc640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9ca40ad870 0x7f9ca41445b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51164/0 (socket says 172.21.15.7:51164) 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca27fc640 1 -- 172.21.15.7:0/2687583775 learned_addr learned my addr 172.21.15.7:0/2687583775 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca37fe640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca4144cc0 0x7f9ca40aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca2ffd640 1 --2- 172.21.15.7:0/2687583775 >> v2:172.21.15.167:3300/0 conn(0x7f9ca4074390 0x7f9ca414aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1592326445 0 0) 0x7f9ca40afed0 con 0x7f9ca40ad870 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c8c003530 con 0x7f9ca40ad870 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2194027358 0 0) 0x7f9c8c003530 con 0x7f9ca40ad870 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca4144cc0 msgr2=0x7f9ca40aebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca4144cc0 0x7f9ca40aebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 >> v2:172.21.15.167:3300/0 conn(0x7f9ca4074390 msgr2=0x7f9ca414aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:08.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 --2- 172.21.15.7:0/2687583775 >> v2:172.21.15.167:3300/0 conn(0x7f9ca4074390 0x7f9ca414aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca40adde0 con 0x7f9ca40ad870 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f9c98001680 con 0x7f9ca40ad870 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 >> v1:172.21.15.7:6789/0 conn(0x7f9ca40ad870 legacy=0x7f9ca41445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca4144cc0 0x7f9c8c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c8c003f40 0x7f9c8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c8c004d50 0x7f9c8c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca40b0140 con 0x7f9c8c003f40 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca4074770 con 0x7f9c8c004d50 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9c8c00bab0 con 0x7f9ca4144cc0 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca27fc640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c8c003f40 0x7f9c8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca2ffd640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca4144cc0 0x7f9c8c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca37fe640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c8c004d50 0x7f9c8c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca27fc640 1 -- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca4144cc0 msgr2=0x7f9c8c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca27fc640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca4144cc0 0x7f9c8c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca27fc640 1 -- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c8c004d50 msgr2=0x7f9c8c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:08.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca27fc640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c8c004d50 0x7f9c8c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca27fc640 1 -- 172.21.15.7:0/2687583775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8c003530 con 0x7f9c8c003f40 2024-03-19T01:24:08.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca2ffd640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca4144cc0 0x7f9c8c0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:08.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.267+0000 7f9ca37fe640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c8c004d50 0x7f9c8c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca27fc640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c8c003f40 0x7f9c8c004810 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800c210 tx=0x7f9c9801cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40819bd6f8371844 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98007d60 con 0x7f9c8c003f40 2024-03-19T01:24:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c980a9020 con 0x7f9c8c003f40 2024-03-19T01:24:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca1ffb640 1 -- 172.21.15.7:0/2687583775 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c980a5bb0 con 0x7f9c8c003f40 2024-03-19T01:24:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 -- 172.21.15.7:0/2687583775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c8c003f40 msgr2=0x7f9c8c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 --2- 172.21.15.7:0/2687583775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c8c003f40 0x7f9c8c004810 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800c210 tx=0x7f9c9801cc70 comp rx=0 tx=0).stop 2024-03-19T01:24:08.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca37fe640 1 -- 172.21.15.7:0/2687583775 reap_dead start 2024-03-19T01:24:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 -- 172.21.15.7:0/2687583775 shutdown_connections 2024-03-19T01:24:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 -- 172.21.15.7:0/2687583775 >> 172.21.15.7:0/2687583775 conn(0x7f9ca406bbb0 msgr2=0x7f9ca40b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 -- 172.21.15.7:0/2687583775 shutdown_connections 2024-03-19T01:24:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 -- 172.21.15.7:0/2687583775 wait complete. 2024-03-19T01:24:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 Processor -- start 2024-03-19T01:24:08.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 -- start start 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c8c003f40 0x7f9ca413d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca4141aa0 0x7f9ca413f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ca413fcc0 0x7f9ca41e3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca40b5a00 con 0x7f9ca413fcc0 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca40b5880 con 0x7f9ca4141aa0 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca9a22640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca41525e0 con 0x7f9c8c003f40 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ca413fcc0 0x7f9ca41e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ca413fcc0 0x7f9ca41e3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48890/0 (socket says 172.21.15.7:48890) 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.271+0000 7f9ca37fe640 1 -- 172.21.15.7:0/3653700237 learned_addr learned my addr 172.21.15.7:0/3653700237 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca2ffd640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c8c003f40 0x7f9ca413d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca27fc640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca4141aa0 0x7f9ca413f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca37fe640 1 -- 172.21.15.7:0/3653700237 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c8c003f40 msgr2=0x7f9ca413d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca37fe640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c8c003f40 0x7f9ca413d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca37fe640 1 -- 172.21.15.7:0/3653700237 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca4141aa0 msgr2=0x7f9ca413f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca37fe640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca4141aa0 0x7f9ca413f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca37fe640 1 -- 172.21.15.7:0/3653700237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca4140c60 con 0x7f9ca413fcc0 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca37fe640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ca413fcc0 0x7f9ca41e3160 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40748f0 tx=0x7f9c9c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9c8bfff640 1 -- 172.21.15.7:0/3653700237 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c01c070 con 0x7f9ca413fcc0 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9c8bfff640 1 -- 172.21.15.7:0/3653700237 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c9c017400 con 0x7f9ca413fcc0 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca4149bf0 con 0x7f9ca413fcc0 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9c8bfff640 1 -- 172.21.15.7:0/3653700237 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c0177d0 con 0x7f9ca413fcc0 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca414a840 con 0x7f9ca413fcc0 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca4075460 con 0x7f9ca413fcc0 2024-03-19T01:24:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9c8bfff640 1 -- 172.21.15.7:0/3653700237 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9c9c01f070 con 0x7f9ca413fcc0 2024-03-19T01:24:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9c8bfff640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9c90067660 0x7f9c90069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9c8bfff640 1 -- 172.21.15.7:0/3653700237 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f9c9c085730 con 0x7f9ca413fcc0 2024-03-19T01:24:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca2ffd640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9c90067660 0x7f9c90069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.275+0000 7f9ca2ffd640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9c90067660 0x7f9c90069b20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009a30 tx=0x7f9c94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:08.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.279+0000 7f9c8bfff640 1 -- 172.21.15.7:0/3653700237 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9c9c012070 con 0x7f9ca413fcc0 2024-03-19T01:24:08.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9ca4147310 con 0x7f9ca413fcc0 2024-03-19T01:24:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9c8bfff640 1 -- 172.21.15.7:0/3653700237 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v26) ==== 84+0+65 (secure 0 0 0) 0x7f9c9c056790 con 0x7f9ca413fcc0 2024-03-19T01:24:08.444 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:08.444 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-03-19T01:24:08.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-19T01:24:08.445 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 1, 2024-03-19T01:24:08.445 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": ".mgr" 2024-03-19T01:24:08.445 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:08.445 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-03-19T01:24:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9c90067660 msgr2=0x7f9c90069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9c90067660 0x7f9c90069b20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009a30 tx=0x7f9c94008040 comp rx=0 tx=0).stop 2024-03-19T01:24:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ca413fcc0 msgr2=0x7f9ca41e3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ca413fcc0 0x7f9ca41e3160 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40748f0 tx=0x7f9c9c00f4a0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 shutdown_connections 2024-03-19T01:24:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9c90067660 0x7f9c90069b20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ca413fcc0 0x7f9ca41e3160 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca4141aa0 0x7f9ca413f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 --2- 172.21.15.7:0/3653700237 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c8c003f40 0x7f9ca413d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 >> 172.21.15.7:0/3653700237 conn(0x7f9ca406bbb0 msgr2=0x7f9ca4146ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:08.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 shutdown_connections 2024-03-19T01:24:08.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.443+0000 7f9ca9a22640 1 -- 172.21.15.7:0/3653700237 wait complete. 2024-03-19T01:24:08.462 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-19T01:24:08.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-19T01:24:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa26a2640 1 Processor -- start 2024-03-19T01:24:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa26a2640 1 -- start start 2024-03-19T01:24:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa26a2640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f8a9c0b6ea0 0x7f8a9c0b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa26a2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a9c0b5640 con 0x7f8a9c0b7930 2024-03-19T01:24:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa26a2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a9c0b7d10 con 0x7f8a9c0b4b30 2024-03-19T01:24:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa26a2640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f8a9c0b6630 con 0x7f8a9c0b6ea0 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8a9c0b7930 0x7f8a9c145370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51166/0 (socket says 172.21.15.7:51166) 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9bfff640 1 -- 172.21.15.7:0/2765374931 learned_addr learned my addr 172.21.15.7:0/2765374931 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506889811 0 0) 0x7f8a9c0b5640 con 0x7f8a9c0b7930 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a88003530 con 0x7f8a9c0b7930 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9b7fe640 1 --2- 172.21.15.7:0/2765374931 >> v2:172.21.15.167:3300/0 conn(0x7f8a9c0b6ea0 0x7f8a9c0b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4046147774 0 0) 0x7f8a88003530 con 0x7f8a9c0b7930 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 >> v2:172.21.15.167:3300/0 conn(0x7f8a9c0b6ea0 msgr2=0x7f8a9c0b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 --2- 172.21.15.7:0/2765374931 >> v2:172.21.15.167:3300/0 conn(0x7f8a9c0b6ea0 0x7f8a9c0b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 >> v1:172.21.15.167:6789/0 conn(0x7f8a9c0b4b30 legacy=0x7f8a9c14fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9c0b58e0 con 0x7f8a9c0b7930 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f8a84001680 con 0x7f8a9c0b7930 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 >> v1:172.21.15.7:6789/0 conn(0x7f8a9c0b7930 legacy=0x7f8a9c145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a88003a10 0x7f8a88003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a88004810 0x7f8a88004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a8800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c0b6630 con 0x7f8a88005220 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8800c170 con 0x7f8a88003a10 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8a8800c370 con 0x7f8a88004810 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9b7fe640 1 --2- 172.21.15.7:0/2765374931 >> v2:172.21.15.167:3300/0 conn(0x7f8a9c0b6ea0 0x7f8a9c0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa0c3f640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a8800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9bfff640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a88003a10 0x7f8a88003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9b7fe640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a88004810 0x7f8a88004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa0c3f640 1 -- 172.21.15.7:0/2765374931 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a88004810 msgr2=0x7f8a88004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa0c3f640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a88004810 0x7f8a88004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa0c3f640 1 -- 172.21.15.7:0/2765374931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a88003a10 msgr2=0x7f8a88003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa0c3f640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a88003a10 0x7f8a88003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8aa0c3f640 1 -- 172.21.15.7:0/2765374931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a88003530 con 0x7f8a88005220 2024-03-19T01:24:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.587+0000 7f8a9bfff640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a88003a10 0x7f8a88003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa0c3f640 1 -- 172.21.15.7:0/2765374931 reap_dead start 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa0c3f640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a8800b9d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8a94001e70 tx=0x7f8a9401c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba941a6db862fc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94001a90 con 0x7f8a88005220 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a94003bc0 con 0x7f8a88005220 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8a9affd640 1 -- 172.21.15.7:0/2765374931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94005020 con 0x7f8a88005220 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2765374931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 msgr2=0x7f8a8800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa26a2640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a8800b9d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8a94001e70 tx=0x7f8a9401c780 comp rx=0 tx=0).stop 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2765374931 shutdown_connections 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa26a2640 1 --2- 172.21.15.7:0/2765374931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a8800b9d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2765374931 >> 172.21.15.7:0/2765374931 conn(0x7f8a9c06b2c0 msgr2=0x7f8a9c06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2765374931 shutdown_connections 2024-03-19T01:24:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2765374931 wait complete. 2024-03-19T01:24:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa26a2640 1 Processor -- start 2024-03-19T01:24:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.591+0000 7f8aa26a2640 1 -- start start 2024-03-19T01:24:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa26a2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a9c1d4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa26a2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a9c0acbc0 0x7f8a9c1d49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa26a2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a9c1de030 0x7f8a9c1de4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa26a2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c072cb0 con 0x7f8a88005220 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa26a2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c072b30 con 0x7f8a9c0acbc0 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa26a2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8a9c0b1bc0 con 0x7f8a9c1de030 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a9c1d4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a9c1d4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48922/0 (socket says 172.21.15.7:48922) 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9bfff640 1 -- 172.21.15.7:0/2579417026 learned_addr learned my addr 172.21.15.7:0/2579417026 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa0c3f640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a9c1de030 0x7f8a9c1de4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9b7fe640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a9c0acbc0 0x7f8a9c1d49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9bfff640 1 -- 172.21.15.7:0/2579417026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a9c1de030 msgr2=0x7f8a9c1de4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9bfff640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a9c1de030 0x7f8a9c1de4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9bfff640 1 -- 172.21.15.7:0/2579417026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a9c0acbc0 msgr2=0x7f8a9c1d49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9bfff640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a9c0acbc0 0x7f8a9c1d49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9bfff640 1 -- 172.21.15.7:0/2579417026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9c0aefb0 con 0x7f8a88005220 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9bfff640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a9c1d4480 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005fd0 tx=0x7f8a8400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a98ff9640 1 -- 172.21.15.7:0/2579417026 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8400c6c0 con 0x7f8a88005220 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a98ff9640 1 -- 172.21.15.7:0/2579417026 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a84018070 con 0x7f8a88005220 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9c1d4f00 con 0x7f8a88005220 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a98ff9640 1 -- 172.21.15.7:0/2579417026 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8400ca60 con 0x7f8a88005220 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a9c0af220 con 0x7f8a88005220 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a9c0ad730 con 0x7f8a88005220 2024-03-19T01:24:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a98ff9640 1 -- 172.21.15.7:0/2579417026 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8a8400cc00 con 0x7f8a88005220 2024-03-19T01:24:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a98ff9640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8a7c067690 0x7f8a7c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a98ff9640 1 -- 172.21.15.7:0/2579417026 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f8a84084d90 con 0x7f8a88005220 2024-03-19T01:24:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.595+0000 7f8a9b7fe640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8a7c067690 0x7f8a7c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.599+0000 7f8a9b7fe640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8a7c067690 0x7f8a7c069b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0bb610 tx=0x7f8a8c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.599+0000 7f8a98ff9640 1 -- 172.21.15.7:0/2579417026 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8a840557a0 con 0x7f8a88005220 2024-03-19T01:24:08.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.755+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8a9c075570 con 0x7f8a88005220 2024-03-19T01:24:08.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.859+0000 7f8a98ff9640 1 -- 172.21.15.7:0/2579417026 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v27) ==== 138+0+0 (secure 0 0 0) 0x7f8a84059840 con 0x7f8a88005220 2024-03-19T01:24:08.863 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' created 2024-03-19T01:24:08.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8a7c067690 msgr2=0x7f8a7c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8a7c067690 0x7f8a7c069b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0bb610 tx=0x7f8a8c008040 comp rx=0 tx=0).stop 2024-03-19T01:24:08.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 msgr2=0x7f8a9c1d4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:08.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a9c1d4480 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005fd0 tx=0x7f8a8400ec10 comp rx=0 tx=0).stop 2024-03-19T01:24:08.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 shutdown_connections 2024-03-19T01:24:08.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8a7c067690 0x7f8a7c069b50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a9c1de030 0x7f8a9c1de4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a9c0acbc0 0x7f8a9c1d49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 --2- 172.21.15.7:0/2579417026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a88005220 0x7f8a9c1d4480 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:08.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 >> 172.21.15.7:0/2579417026 conn(0x7f8a9c06b2c0 msgr2=0x7f8a9c0b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:08.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 shutdown_connections 2024-03-19T01:24:08.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:08.863+0000 7f8aa26a2640 1 -- 172.21.15.7:0/2579417026 wait complete. 2024-03-19T01:24:08.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-19T01:24:09.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468feb6640 1 Processor -- start 2024-03-19T01:24:09.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468feb6640 1 -- start start 2024-03-19T01:24:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468feb6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 0x7f46881450d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468feb6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f468806ff90 0x7f4688071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468feb6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46880b5d80 con 0x7f4688072440 2024-03-19T01:24:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468feb6640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f4688145610 con 0x7f468806ff90 2024-03-19T01:24:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468feb6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46880704c0 con 0x7f46880b59a0 2024-03-19T01:24:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468d451640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f468806ff90 0x7f4688071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468d451640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f468806ff90 0x7f4688071f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48938/0 (socket says 172.21.15.7:48938) 2024-03-19T01:24:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468d451640 1 -- 172.21.15.7:0/3532715670 learned_addr learned my addr 172.21.15.7:0/3532715670 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468dc52640 1 --2- 172.21.15.7:0/3532715670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 0x7f46881450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468d451640 1 -- 172.21.15.7:0/3532715670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 msgr2=0x7f46881450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468d451640 1 --2- 172.21.15.7:0/3532715670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 0x7f46881450d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468d451640 1 -- 172.21.15.7:0/3532715670 >> v1:172.21.15.7:6789/0 conn(0x7f4688072440 legacy=0x7f46880b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.023+0000 7f468d451640 1 -- 172.21.15.7:0/3532715670 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4688072a80 con 0x7f468806ff90 2024-03-19T01:24:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468d451640 1 --2- 172.21.15.7:0/3532715670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 0x7f4688071f00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4684009e50 tx=0x7f468401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec63ebd776ce256 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468cc50640 1 -- 172.21.15.7:0/3532715670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840a55c0 con 0x7f468806ff90 2024-03-19T01:24:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468cc50640 1 -- 172.21.15.7:0/3532715670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46840a5760 con 0x7f468806ff90 2024-03-19T01:24:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468cc50640 1 -- 172.21.15.7:0/3532715670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840a5b30 con 0x7f468806ff90 2024-03-19T01:24:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 -- 172.21.15.7:0/3532715670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 msgr2=0x7f4688071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 --2- 172.21.15.7:0/3532715670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 0x7f4688071f00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4684009e50 tx=0x7f468401c910 comp rx=0 tx=0).stop 2024-03-19T01:24:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 -- 172.21.15.7:0/3532715670 shutdown_connections 2024-03-19T01:24:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 --2- 172.21.15.7:0/3532715670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 0x7f4688071f00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 --2- 172.21.15.7:0/3532715670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 0x7f46881450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 -- 172.21.15.7:0/3532715670 >> 172.21.15.7:0/3532715670 conn(0x7f468806c3b0 msgr2=0x7f468806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 -- 172.21.15.7:0/3532715670 shutdown_connections 2024-03-19T01:24:09.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 -- 172.21.15.7:0/3532715670 wait complete. 2024-03-19T01:24:09.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 Processor -- start 2024-03-19T01:24:09.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.027+0000 7f468feb6640 1 -- start start 2024-03-19T01:24:09.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468feb6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 0x7f46881442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468feb6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4688072440 0x7f4688147d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468feb6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 0x7f4688132a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468feb6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f46880b4f50 con 0x7f468806ff90 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468feb6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46880b4dd0 con 0x7f4688072440 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468feb6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4688152a70 con 0x7f46880b59a0 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468dc52640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 0x7f46881442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468dc52640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 0x7f46881442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48952/0 (socket says 172.21.15.7:48952) 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468dc52640 1 -- 172.21.15.7:0/3837503277 learned_addr learned my addr 172.21.15.7:0/3837503277 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468e453640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 0x7f4688132a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468dc52640 1 -- 172.21.15.7:0/3837503277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 msgr2=0x7f4688132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468d451640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4688072440 0x7f4688147d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468dc52640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 0x7f4688132a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468dc52640 1 -- 172.21.15.7:0/3837503277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4688072440 msgr2=0x7f4688147d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468dc52640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4688072440 0x7f4688147d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468dc52640 1 -- 172.21.15.7:0/3837503277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468813ea10 con 0x7f468806ff90 2024-03-19T01:24:09.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468dc52640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 0x7f46881442f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f467c00d5f0 tx=0x7f467c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:09.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f467a7fc640 1 -- 172.21.15.7:0/3837503277 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c0125f0 con 0x7f468806ff90 2024-03-19T01:24:09.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f467a7fc640 1 -- 172.21.15.7:0/3837503277 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f467c019070 con 0x7f468806ff90 2024-03-19T01:24:09.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f468813e580 con 0x7f468806ff90 2024-03-19T01:24:09.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f467a7fc640 1 -- 172.21.15.7:0/3837503277 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c0129b0 con 0x7f468806ff90 2024-03-19T01:24:09.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f468813c770 con 0x7f468806ff90 2024-03-19T01:24:09.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4688075570 con 0x7f468806ff90 2024-03-19T01:24:09.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f467a7fc640 1 -- 172.21.15.7:0/3837503277 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f467c012b50 con 0x7f468806ff90 2024-03-19T01:24:09.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f467a7fc640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f466c067660 0x7f466c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:09.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f467a7fc640 1 -- 172.21.15.7:0/3837503277 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5227+0+0 (secure 0 0 0) 0x7f467c086560 con 0x7f468806ff90 2024-03-19T01:24:09.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.031+0000 7f468d451640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f466c067660 0x7f466c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:09.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.035+0000 7f468d451640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f466c067660 0x7f466c069b20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46840ae000 tx=0x7f46840a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:09.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.035+0000 7f467a7fc640 1 -- 172.21.15.7:0/3837503277 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f467c0574a0 con 0x7f468806ff90 2024-03-19T01:24:09.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.191+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f468813e2a0 con 0x7f468806ff90 2024-03-19T01:24:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.871+0000 7f467a7fc640 1 -- 172.21.15.7:0/3837503277 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v28) ==== 170+0+0 (secure 0 0 0) 0x7f467c056da0 con 0x7f468806ff90 2024-03-19T01:24:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' created 2024-03-19T01:24:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f466c067660 msgr2=0x7f466c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f466c067660 0x7f466c069b20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46840ae000 tx=0x7f46840a7070 comp rx=0 tx=0).stop 2024-03-19T01:24:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 msgr2=0x7f46881442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 0x7f46881442f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f467c00d5f0 tx=0x7f467c00dab0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 shutdown_connections 2024-03-19T01:24:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f466c067660 0x7f466c069b20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46880b59a0 0x7f4688132a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4688072440 0x7f4688147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 --2- 172.21.15.7:0/3837503277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f468806ff90 0x7f46881442f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:09.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 >> 172.21.15.7:0/3837503277 conn(0x7f468806c3b0 msgr2=0x7f46880b6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:09.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 shutdown_connections 2024-03-19T01:24:09.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:09.875+0000 7f468feb6640 1 -- 172.21.15.7:0/3837503277 wait complete. 2024-03-19T01:24:09.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-19T01:24:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.035+0000 7f2182f3e640 1 Processor -- start 2024-03-19T01:24:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2182f3e640 1 -- start start 2024-03-19T01:24:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2182f3e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c0bd400 0x7f217c0b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2182f3e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f217c141650 0x7f217c14ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2182f3e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f217c0bd7e0 con 0x7f217c14a960 2024-03-19T01:24:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2182f3e640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f217c0b4f20 con 0x7f217c141650 2024-03-19T01:24:10.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2182f3e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f217c0b6d70 con 0x7f217c0bd400 2024-03-19T01:24:10.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f21814db640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f217c141650 0x7f217c14ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:10.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2180cda640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c0bd400 0x7f217c0b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:10.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f217bfff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f217c14a960 0x7f217c140f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:56946/0 (socket says 172.21.15.7:56946) 2024-03-19T01:24:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f217bfff640 1 -- 172.21.15.7:0/2673860153 learned_addr learned my addr 172.21.15.7:0/2673860153 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f217b7fe640 1 -- 172.21.15.7:0/2673860153 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 616555325 0 0) 0x7f217c0bd7e0 con 0x7f217c14a960 2024-03-19T01:24:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f217b7fe640 1 -- 172.21.15.7:0/2673860153 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2164003530 con 0x7f217c14a960 2024-03-19T01:24:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2180cda640 1 -- 172.21.15.7:0/2673860153 >> v2:172.21.15.167:3300/0 conn(0x7f217c141650 msgr2=0x7f217c14ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2180cda640 1 --2- 172.21.15.7:0/2673860153 >> v2:172.21.15.167:3300/0 conn(0x7f217c141650 0x7f217c14ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2180cda640 1 -- 172.21.15.7:0/2673860153 >> v1:172.21.15.167:6789/0 conn(0x7f217c14a960 legacy=0x7f217c140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:10.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.039+0000 7f2180cda640 1 -- 172.21.15.7:0/2673860153 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c0b5ce0 con 0x7f217c0bd400 2024-03-19T01:24:10.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2180cda640 1 --2- 172.21.15.7:0/2673860153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c0bd400 0x7f217c0b57a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f216c009d90 tx=0x7f216c01c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e306a0f7844ce90d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:10.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f217b7fe640 1 -- 172.21.15.7:0/2673860153 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c0b2070 con 0x7f217c0bd400 2024-03-19T01:24:10.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f217b7fe640 1 -- 172.21.15.7:0/2673860153 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f216c0a5890 con 0x7f217c0bd400 2024-03-19T01:24:10.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f217b7fe640 1 -- 172.21.15.7:0/2673860153 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c0a5c60 con 0x7f217c0bd400 2024-03-19T01:24:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 -- 172.21.15.7:0/2673860153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c0bd400 msgr2=0x7f217c0b57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 --2- 172.21.15.7:0/2673860153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c0bd400 0x7f217c0b57a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f216c009d90 tx=0x7f216c01c8e0 comp rx=0 tx=0).stop 2024-03-19T01:24:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 -- 172.21.15.7:0/2673860153 shutdown_connections 2024-03-19T01:24:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 --2- 172.21.15.7:0/2673860153 >> v2:172.21.15.167:3300/0 conn(0x7f217c141650 0x7f217c14ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 --2- 172.21.15.7:0/2673860153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c0bd400 0x7f217c0b57a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 -- 172.21.15.7:0/2673860153 >> 172.21.15.7:0/2673860153 conn(0x7f217c072df0 msgr2=0x7f217c073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 -- 172.21.15.7:0/2673860153 shutdown_connections 2024-03-19T01:24:10.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 -- 172.21.15.7:0/2673860153 wait complete. 2024-03-19T01:24:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 Processor -- start 2024-03-19T01:24:10.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 -- start start 2024-03-19T01:24:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f217c0bd400 0x7f217c148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:10.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217c141650 0x7f217c13ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c14a960 0x7f217c145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f217c154510 con 0x7f217c141650 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f217c0b4a80 con 0x7f217c0bd400 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2182f3e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f217c154690 con 0x7f217c14a960 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f217bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217c141650 0x7f217c13ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f217bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217c141650 0x7f217c13ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48958/0 (socket says 172.21.15.7:48958) 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f217bfff640 1 -- 172.21.15.7:0/215537129 learned_addr learned my addr 172.21.15.7:0/215537129 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f21814db640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c14a960 0x7f217c145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f2180cda640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f217c0bd400 0x7f217c148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f217bfff640 1 -- 172.21.15.7:0/215537129 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c14a960 msgr2=0x7f217c145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.043+0000 7f217bfff640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c14a960 0x7f217c145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f217bfff640 1 -- 172.21.15.7:0/215537129 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f217c0bd400 msgr2=0x7f217c148200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f217bfff640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f217c0bd400 0x7f217c148200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:10.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f217bfff640 1 -- 172.21.15.7:0/215537129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c144b30 con 0x7f217c141650 2024-03-19T01:24:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f2180cda640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f217c0bd400 0x7f217c148200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f217bfff640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217c141650 0x7f217c13ff20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f217000c3c0 tx=0x7f217000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f21797fa640 1 -- 172.21.15.7:0/215537129 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170017070 con 0x7f217c141650 2024-03-19T01:24:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f21797fa640 1 -- 172.21.15.7:0/215537129 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2170005cf0 con 0x7f217c141650 2024-03-19T01:24:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f21797fa640 1 -- 172.21.15.7:0/215537129 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21700072f0 con 0x7f217c141650 2024-03-19T01:24:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217c1dfed0 con 0x7f217c141650 2024-03-19T01:24:10.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f217c13d9e0 con 0x7f217c141650 2024-03-19T01:24:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f217c075570 con 0x7f217c141650 2024-03-19T01:24:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f21797fa640 1 -- 172.21.15.7:0/215537129 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f217001a070 con 0x7f217c141650 2024-03-19T01:24:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f21797fa640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2168067660 0x7f2168069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f21797fa640 1 -- 172.21.15.7:0/215537129 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5609+0+0 (secure 0 0 0) 0x7f2170084f50 con 0x7f217c141650 2024-03-19T01:24:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f2180cda640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2168067660 0x7f2168069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.047+0000 7f2180cda640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2168067660 0x7f2168069b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f216c0af000 tx=0x7f216c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:10.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.051+0000 7f21797fa640 1 -- 172.21.15.7:0/215537129 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2170021020 con 0x7f217c141650 2024-03-19T01:24:10.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.247+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f217c13c260 con 0x7f217c141650 2024-03-19T01:24:10.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.875+0000 7f21797fa640 1 -- 172.21.15.7:0/215537129 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7f2170055690 con 0x7f217c141650 2024-03-19T01:24:10.877 INFO:teuthology.orchestra.run.smithi007.stderr:new fs with metadata pool 4 and data pool 5 2024-03-19T01:24:10.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2168067660 msgr2=0x7f2168069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:10.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2168067660 0x7f2168069b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f216c0af000 tx=0x7f216c0a7070 comp rx=0 tx=0).stop 2024-03-19T01:24:10.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217c141650 msgr2=0x7f217c13ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:10.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217c141650 0x7f217c13ff20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f217000c3c0 tx=0x7f217000c880 comp rx=0 tx=0).stop 2024-03-19T01:24:10.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 shutdown_connections 2024-03-19T01:24:10.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2168067660 0x7f2168069b20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:10.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f217c14a960 0x7f217c145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:10.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217c141650 0x7f217c13ff20 secure :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f217000c3c0 tx=0x7f217000c880 comp rx=0 tx=0).stop 2024-03-19T01:24:10.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 --2- 172.21.15.7:0/215537129 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f217c0bd400 0x7f217c148200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:10.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 >> 172.21.15.7:0/215537129 conn(0x7f217c072df0 msgr2=0x7f217c13c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:10.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.883+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 shutdown_connections 2024-03-19T01:24:10.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:10.887+0000 7f2182f3e640 1 -- 172.21.15.7:0/215537129 wait complete. 2024-03-19T01:24:10.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:11.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c9d87640 1 Processor -- start 2024-03-19T01:24:11.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c9d87640 1 -- start start 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c9d87640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f18c4071500 0x7f18c4070210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c9d87640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c413e6c0 0x7f18c414ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c9d87640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c4071060 con 0x7f18c41476e0 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c9d87640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f18c40b9b40 con 0x7f18c4071500 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c9d87640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f18c4070750 con 0x7f18c413e6c0 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18bbfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f18c41476e0 0x7f18c413dfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51188/0 (socket says 172.21.15.7:51188) 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18bbfff640 1 -- 172.21.15.7:0/128631906 learned_addr learned my addr 172.21.15.7:0/128631906 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c37fe640 1 --2- 172.21.15.7:0/128631906 >> v2:172.21.15.167:3300/0 conn(0x7f18c4071500 0x7f18c4070210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c3fff640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c413e6c0 0x7f18c414ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1738360762 0 0) 0x7f18c4071060 con 0x7f18c41476e0 2024-03-19T01:24:11.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18b0003530 con 0x7f18c41476e0 2024-03-19T01:24:11.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3151912942 0 0) 0x7f18b0003530 con 0x7f18c41476e0 2024-03-19T01:24:11.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c413e6c0 msgr2=0x7f18c414ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:11.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c413e6c0 0x7f18c414ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 >> v2:172.21.15.167:3300/0 conn(0x7f18c4071500 msgr2=0x7f18c4070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:11.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 --2- 172.21.15.7:0/128631906 >> v2:172.21.15.167:3300/0 conn(0x7f18c4071500 0x7f18c4070210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c4070d80 con 0x7f18c41476e0 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f18ac001680 con 0x7f18c41476e0 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c37fe640 1 --2- 172.21.15.7:0/128631906 >> v2:172.21.15.167:3300/0 conn(0x7f18c4071500 0x7f18c4070210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 >> v1:172.21.15.7:6789/0 conn(0x7f18c41476e0 legacy=0x7f18c413dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b00039c0 0x7f18b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c413e6c0 0x7f18b0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18b0004d50 0x7f18b00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f18c40b9b40 con 0x7f18b00039c0 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f18c4070750 con 0x7f18b0004d50 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f18b000ba80 con 0x7f18c413e6c0 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c37fe640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b00039c0 0x7f18b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18bbfff640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c413e6c0 0x7f18b0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c3fff640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18b0004d50 0x7f18b00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c37fe640 1 -- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c413e6c0 msgr2=0x7f18b0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:11.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c37fe640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c413e6c0 0x7f18b0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c37fe640 1 -- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18b0004d50 msgr2=0x7f18b00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:11.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c37fe640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18b0004d50 0x7f18b00051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18c37fe640 1 -- 172.21.15.7:0/128631906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b0003530 con 0x7f18b00039c0 2024-03-19T01:24:11.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.043+0000 7f18bbfff640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c413e6c0 0x7f18b0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:11.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c37fe640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b00039c0 0x7f18b0003de0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009cb0 tx=0x7f18b401cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5126c1f54a760b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:11.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b40b2070 con 0x7f18b00039c0 2024-03-19T01:24:11.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b40a5730 con 0x7f18b00039c0 2024-03-19T01:24:11.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c2ffd640 1 -- 172.21.15.7:0/128631906 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4009880 con 0x7f18b00039c0 2024-03-19T01:24:11.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c9d87640 1 -- 172.21.15.7:0/128631906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b00039c0 msgr2=0x7f18b0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c9d87640 1 --2- 172.21.15.7:0/128631906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b00039c0 0x7f18b0003de0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009cb0 tx=0x7f18b401cc90 comp rx=0 tx=0).stop 2024-03-19T01:24:11.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c3fff640 1 -- 172.21.15.7:0/128631906 reap_dead start 2024-03-19T01:24:11.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c9d87640 1 -- 172.21.15.7:0/128631906 shutdown_connections 2024-03-19T01:24:11.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c9d87640 1 -- 172.21.15.7:0/128631906 >> 172.21.15.7:0/128631906 conn(0x7f18c4074130 msgr2=0x7f18c4074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:11.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c9d87640 1 -- 172.21.15.7:0/128631906 shutdown_connections 2024-03-19T01:24:11.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c9d87640 1 -- 172.21.15.7:0/128631906 wait complete. 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.047+0000 7f18c9d87640 1 Processor -- start 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 -- start start 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18b00039c0 0x7f18c413b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18c40b3660 0x7f18c40b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c4144d00 0x7f18c40b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f18c4147ae0 con 0x7f18c40b3660 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f18c40ba1c0 con 0x7f18b00039c0 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f18c4147c60 con 0x7f18c4144d00 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18bbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18c40b3660 0x7f18c40b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18bbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18c40b3660 0x7f18c40b27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48978/0 (socket says 172.21.15.7:48978) 2024-03-19T01:24:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18bbfff640 1 -- 172.21.15.7:0/2098614068 learned_addr learned my addr 172.21.15.7:0/2098614068 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c37fe640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18b00039c0 0x7f18c413b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18bbfff640 1 -- 172.21.15.7:0/2098614068 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c4144d00 msgr2=0x7f18c40b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18bbfff640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c4144d00 0x7f18c40b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18bbfff640 1 -- 172.21.15.7:0/2098614068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18b00039c0 msgr2=0x7f18c413b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18bbfff640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18b00039c0 0x7f18c413b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18bbfff640 1 -- 172.21.15.7:0/2098614068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c40b1640 con 0x7f18c40b3660 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18bbfff640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18c40b3660 0x7f18c40b27b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f18ac001210 tx=0x7f18ac00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c0ff9640 1 -- 172.21.15.7:0/2098614068 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac019070 con 0x7f18c40b3660 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c0ff9640 1 -- 172.21.15.7:0/2098614068 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18ac0072d0 con 0x7f18c40b3660 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c40b1920 con 0x7f18c40b3660 2024-03-19T01:24:11.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c0ff9640 1 -- 172.21.15.7:0/2098614068 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac007680 con 0x7f18c40b3660 2024-03-19T01:24:11.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c4135c00 con 0x7f18c40b3660 2024-03-19T01:24:11.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c40b1430 con 0x7f18c40b3660 2024-03-19T01:24:11.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c0ff9640 1 -- 172.21.15.7:0/2098614068 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f18ac01c070 con 0x7f18c40b3660 2024-03-19T01:24:11.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c0ff9640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f18a4067660 0x7f18a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c37fe640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f18a4067660 0x7f18a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.051+0000 7f18c0ff9640 1 -- 172.21.15.7:0/2098614068 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5709+0+0 (secure 0 0 0) 0x7f18ac0853c0 con 0x7f18c40b3660 2024-03-19T01:24:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.055+0000 7f18c37fe640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f18a4067660 0x7f18a4069b20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f18b40af000 tx=0x7f18b40046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:11.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.059+0000 7f18c0ff9640 1 -- 172.21.15.7:0/2098614068 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f18ac0560b0 con 0x7f18c40b3660 2024-03-19T01:24:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.214+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18c4075460 con 0x7f18c40b3660 2024-03-19T01:24:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.214+0000 7f18c0ff9640 1 -- 172.21.15.7:0/2098614068 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+16646 (secure 0 0 0) 0x7f18ac0559b0 con 0x7f18c40b3660 2024-03-19T01:24:11.217 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:11.217 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":29,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:24:10.251068+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:24:08.795930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-19T01:24:09.233323+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:11.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.214+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f18a4067660 msgr2=0x7f18a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.214+0000 7f18c9d87640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f18a4067660 0x7f18a4069b20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f18b40af000 tx=0x7f18b40046d0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.214+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18c40b3660 msgr2=0x7f18c40b27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.214+0000 7f18c9d87640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18c40b3660 0x7f18c40b27b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f18ac001210 tx=0x7f18ac00db90 comp rx=0 tx=0).stop 2024-03-19T01:24:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.214+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 shutdown_connections 2024-03-19T01:24:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.218+0000 7f18c9d87640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f18a4067660 0x7f18a4069b20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.218+0000 7f18c9d87640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18c4144d00 0x7f18c40b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.218+0000 7f18c9d87640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18c40b3660 0x7f18c40b27b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.218+0000 7f18c9d87640 1 --2- 172.21.15.7:0/2098614068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18b00039c0 0x7f18c413b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.218+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 >> 172.21.15.7:0/2098614068 conn(0x7f18c4074130 msgr2=0x7f18c413c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.218+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 shutdown_connections 2024-03-19T01:24:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.218+0000 7f18c9d87640 1 -- 172.21.15.7:0/2098614068 wait complete. 2024-03-19T01:24:11.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:11.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd7424b2640 1 Processor -- start 2024-03-19T01:24:11.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd7424b2640 1 -- start start 2024-03-19T01:24:11.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd7424b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b1310 0x7fd73c13d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd7424b2640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fd73c0b16f0 0x7fd73c13dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd7424b2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd73c0b8140 0x7fd73c143830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd7424b2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fd73c0b2ca0 con 0x7fd73c0b8140 2024-03-19T01:24:11.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd7424b2640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fd73c0b7080 con 0x7fd73c0b16f0 2024-03-19T01:24:11.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd7424b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd73c0b29c0 con 0x7fd73c0b1310 2024-03-19T01:24:11.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd740a4f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd73c0b8140 0x7fd73c143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd740a4f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd73c0b8140 0x7fd73c143830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48980/0 (socket says 172.21.15.7:48980) 2024-03-19T01:24:11.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd740a4f640 1 -- 172.21.15.7:0/3785081449 learned_addr learned my addr 172.21.15.7:0/3785081449 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:11.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd73bfff640 1 --2- 172.21.15.7:0/3785081449 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b1310 0x7fd73c13d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd73b7fe640 1 --2- 172.21.15.7:0/3785081449 >> v2:172.21.15.167:3300/0 conn(0x7fd73c0b16f0 0x7fd73c13dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd740a4f640 1 -- 172.21.15.7:0/3785081449 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b1310 msgr2=0x7fd73c13d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:11.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd740a4f640 1 --2- 172.21.15.7:0/3785081449 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b1310 0x7fd73c13d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd740a4f640 1 -- 172.21.15.7:0/3785081449 >> v2:172.21.15.167:3300/0 conn(0x7fd73c0b16f0 msgr2=0x7fd73c13dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:11.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd740a4f640 1 --2- 172.21.15.7:0/3785081449 >> v2:172.21.15.167:3300/0 conn(0x7fd73c0b16f0 0x7fd73c13dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd740a4f640 1 -- 172.21.15.7:0/3785081449 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c0b7320 con 0x7fd73c0b8140 2024-03-19T01:24:11.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd73b7fe640 1 --2- 172.21.15.7:0/3785081449 >> v2:172.21.15.167:3300/0 conn(0x7fd73c0b16f0 0x7fd73c13dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:11.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd740a4f640 1 --2- 172.21.15.7:0/3785081449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b8140 0x7fd73c143830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd73c06dcb0 tx=0x7fd73401d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70acba5575495909 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd73affd640 1 -- 172.21.15.7:0/3785081449 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73400a2d0 con 0x7fd73c0b8140 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd73affd640 1 -- 172.21.15.7:0/3785081449 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7340a67e0 con 0x7fd73c0b8140 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.354+0000 7fd73affd640 1 -- 172.21.15.7:0/3785081449 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7340a6bb0 con 0x7fd73c0b8140 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- 172.21.15.7:0/3785081449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b8140 msgr2=0x7fd73c143830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3785081449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b8140 0x7fd73c143830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd73c06dcb0 tx=0x7fd73401d9a0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- 172.21.15.7:0/3785081449 shutdown_connections 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3785081449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b8140 0x7fd73c143830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3785081449 >> v2:172.21.15.167:3300/0 conn(0x7fd73c0b16f0 0x7fd73c13dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3785081449 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b1310 0x7fd73c13d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- 172.21.15.7:0/3785081449 >> 172.21.15.7:0/3785081449 conn(0x7fd73c0b5330 msgr2=0x7fd73c0b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- 172.21.15.7:0/3785081449 shutdown_connections 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- 172.21.15.7:0/3785081449 wait complete. 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 Processor -- start 2024-03-19T01:24:11.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- start start 2024-03-19T01:24:11.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd73c0b1310 0x7fd73c13f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b16f0 0x7fd73c1ddfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b8140 0x7fd73c1de4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c0afeb0 con 0x7fd73c0b16f0 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c0afd30 con 0x7fd73c0b1310 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd73c0b0030 con 0x7fd73c0b8140 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b16f0 0x7fd73c1ddfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b16f0 0x7fd73c1ddfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48990/0 (socket says 172.21.15.7:48990) 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73b7fe640 1 -- 172.21.15.7:0/3372781208 learned_addr learned my addr 172.21.15.7:0/3372781208 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73bfff640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd73c0b1310 0x7fd73c13f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd740a4f640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b8140 0x7fd73c1de4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73b7fe640 1 -- 172.21.15.7:0/3372781208 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b8140 msgr2=0x7fd73c1de4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73b7fe640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b8140 0x7fd73c1de4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73b7fe640 1 -- 172.21.15.7:0/3372781208 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd73c0b1310 msgr2=0x7fd73c13f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73b7fe640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd73c0b1310 0x7fd73c13f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73b7fe640 1 -- 172.21.15.7:0/3372781208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c0a9c70 con 0x7fd73c0b16f0 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73bfff640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd73c0b1310 0x7fd73c13f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd73b7fe640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b16f0 0x7fd73c1ddfb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd73000c7f0 tx=0x7fd73000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd738ff9640 1 -- 172.21.15.7:0/3372781208 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd730007a30 con 0x7fd73c0b16f0 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd738ff9640 1 -- 172.21.15.7:0/3372781208 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd730007bd0 con 0x7fd73c0b16f0 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73c141020 con 0x7fd73c0b16f0 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd738ff9640 1 -- 172.21.15.7:0/3372781208 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7300113f0 con 0x7fd73c0b16f0 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.358+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd73c14dc60 con 0x7fd73c0b16f0 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.362+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd73c075460 con 0x7fd73c0b16f0 2024-03-19T01:24:11.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.362+0000 7fd738ff9640 1 -- 172.21.15.7:0/3372781208 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd730011590 con 0x7fd73c0b16f0 2024-03-19T01:24:11.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.362+0000 7fd738ff9640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd728067660 0x7fd728069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.362+0000 7fd738ff9640 1 -- 172.21.15.7:0/3372781208 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5709+0+0 (secure 0 0 0) 0x7fd730085a30 con 0x7fd73c0b16f0 2024-03-19T01:24:11.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.362+0000 7fd73bfff640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd728067660 0x7fd728069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.362+0000 7fd73bfff640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd728067660 0x7fd728069b20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00a990 tx=0x7fd72c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:11.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.366+0000 7fd738ff9640 1 -- 172.21.15.7:0/3372781208 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd730015020 con 0x7fd73c0b16f0 2024-03-19T01:24:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.526+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd73c071bf0 con 0x7fd73c0b16f0 2024-03-19T01:24:11.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.526+0000 7fd738ff9640 1 -- 172.21.15.7:0/3372781208 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+16646 (secure 0 0 0) 0x7fd7300566a0 con 0x7fd73c0b16f0 2024-03-19T01:24:11.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:11.530 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":29,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:24:10.251068+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:24:08.795930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-19T01:24:09.233323+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd728067660 msgr2=0x7fd728069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd728067660 0x7fd728069b20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00a990 tx=0x7fd72c008040 comp rx=0 tx=0).stop 2024-03-19T01:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b16f0 msgr2=0x7fd73c1ddfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b16f0 0x7fd73c1ddfb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd73000c7f0 tx=0x7fd73000ccb0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 shutdown_connections 2024-03-19T01:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd728067660 0x7fd728069b20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd73c0b8140 0x7fd73c1de4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd73c0b16f0 0x7fd73c1ddfb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 --2- 172.21.15.7:0/3372781208 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd73c0b1310 0x7fd73c13f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 >> 172.21.15.7:0/3372781208 conn(0x7fd73c0b5330 msgr2=0x7fd73c1417a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:11.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 shutdown_connections 2024-03-19T01:24:11.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.530+0000 7fd7424b2640 1 -- 172.21.15.7:0/3372781208 wait complete. 2024-03-19T01:24:11.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-19T01:24:11.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b8d2e6640 1 Processor -- start 2024-03-19T01:24:11.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b8d2e6640 1 -- start start 2024-03-19T01:24:11.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b8d2e6640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b880b4600 0x7f6b8814ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b8d2e6640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f6b880bd6a0 0x7f6b8814b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b8d2e6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b880b49e0 con 0x7f6b881410e0 2024-03-19T01:24:11.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b8d2e6640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f6b880b5510 con 0x7f6b880b4600 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b8d2e6640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f6b880b57f0 con 0x7f6b880bd6a0 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b86ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b880b4600 0x7f6b8814ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b877fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6b881410e0 0x7f6b88141570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51204/0 (socket says 172.21.15.7:51204) 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b877fe640 1 -- 172.21.15.7:0/1556959199 learned_addr learned my addr 172.21.15.7:0/1556959199 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b867fc640 1 --2- 172.21.15.7:0/1556959199 >> v2:172.21.15.167:3300/0 conn(0x7f6b880bd6a0 0x7f6b8814b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b86ffd640 1 -- 172.21.15.7:0/1556959199 >> v2:172.21.15.167:3300/0 conn(0x7f6b880bd6a0 msgr2=0x7f6b8814b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b86ffd640 1 --2- 172.21.15.7:0/1556959199 >> v2:172.21.15.167:3300/0 conn(0x7f6b880bd6a0 0x7f6b8814b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b86ffd640 1 -- 172.21.15.7:0/1556959199 >> v1:172.21.15.7:6789/0 conn(0x7f6b881410e0 legacy=0x7f6b88141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b86ffd640 1 -- 172.21.15.7:0/1556959199 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b880b5ad0 con 0x7f6b880b4600 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b85ffb640 1 -- 172.21.15.7:0/1556959199 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1725824761 0 0) 0x7f6b880b49e0 con 0x7f6b881410e0 2024-03-19T01:24:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.662+0000 7f6b867fc640 1 --2- 172.21.15.7:0/1556959199 >> v2:172.21.15.167:3300/0 conn(0x7f6b880bd6a0 0x7f6b8814b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:11.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b86ffd640 1 --2- 172.21.15.7:0/1556959199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880b4600 0x7f6b8814ab30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6b880b7350 tx=0x7f6b7801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9debf52ce4ad538a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:11.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b85ffb640 1 -- 172.21.15.7:0/1556959199 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b780b0070 con 0x7f6b880b4600 2024-03-19T01:24:11.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b85ffb640 1 -- 172.21.15.7:0/1556959199 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b780ab600 con 0x7f6b880b4600 2024-03-19T01:24:11.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b85ffb640 1 -- 172.21.15.7:0/1556959199 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b780ab9d0 con 0x7f6b880b4600 2024-03-19T01:24:11.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1556959199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880b4600 msgr2=0x7f6b8814ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 --2- 172.21.15.7:0/1556959199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880b4600 0x7f6b8814ab30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6b880b7350 tx=0x7f6b7801c360 comp rx=0 tx=0).stop 2024-03-19T01:24:11.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1556959199 shutdown_connections 2024-03-19T01:24:11.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 --2- 172.21.15.7:0/1556959199 >> v2:172.21.15.167:3300/0 conn(0x7f6b880bd6a0 0x7f6b8814b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 --2- 172.21.15.7:0/1556959199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880b4600 0x7f6b8814ab30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1556959199 >> 172.21.15.7:0/1556959199 conn(0x7f6b88072df0 msgr2=0x7f6b88073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:11.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1556959199 shutdown_connections 2024-03-19T01:24:11.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1556959199 wait complete. 2024-03-19T01:24:11.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 Processor -- start 2024-03-19T01:24:11.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- start start 2024-03-19T01:24:11.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b880b4600 0x7f6b8814a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880bd6a0 0x7f6b8813a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b881410e0 0x7f6b881351b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6b88154740 con 0x7f6b880bd6a0 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6b881548c0 con 0x7f6b880b4600 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6b88154a40 con 0x7f6b881410e0 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b867fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880bd6a0 0x7f6b8813a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b867fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880bd6a0 0x7f6b8813a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48996/0 (socket says 172.21.15.7:48996) 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b867fc640 1 -- 172.21.15.7:0/1870106120 learned_addr learned my addr 172.21.15.7:0/1870106120 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b867fc640 1 -- 172.21.15.7:0/1870106120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b881410e0 msgr2=0x7f6b881351b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b867fc640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b881410e0 0x7f6b881351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b867fc640 1 -- 172.21.15.7:0/1870106120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b880b4600 msgr2=0x7f6b8814a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b867fc640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b880b4600 0x7f6b8814a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b867fc640 1 -- 172.21.15.7:0/1870106120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b88142d10 con 0x7f6b880bd6a0 2024-03-19T01:24:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b867fc640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880bd6a0 0x7f6b8813a070 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00e810 tx=0x7f6b7c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b6f7fe640 1 -- 172.21.15.7:0/1870106120 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c017070 con 0x7f6b880bd6a0 2024-03-19T01:24:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b6f7fe640 1 -- 172.21.15.7:0/1870106120 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7c00c8d0 con 0x7f6b880bd6a0 2024-03-19T01:24:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.666+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8813bd40 con 0x7f6b880bd6a0 2024-03-19T01:24:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.670+0000 7f6b6f7fe640 1 -- 172.21.15.7:0/1870106120 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c00f040 con 0x7f6b880bd6a0 2024-03-19T01:24:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.670+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b88148550 con 0x7f6b880bd6a0 2024-03-19T01:24:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.670+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b88075570 con 0x7f6b880bd6a0 2024-03-19T01:24:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.670+0000 7f6b6f7fe640 1 -- 172.21.15.7:0/1870106120 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6b7c01a070 con 0x7f6b880bd6a0 2024-03-19T01:24:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.670+0000 7f6b6f7fe640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b74067660 0x7f6b74069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.670+0000 7f6b6f7fe640 1 -- 172.21.15.7:0/1870106120 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5709+0+0 (secure 0 0 0) 0x7f6b7c085410 con 0x7f6b880bd6a0 2024-03-19T01:24:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.670+0000 7f6b86ffd640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b74067660 0x7f6b74069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.670+0000 7f6b86ffd640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b74067660 0x7f6b74069b20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b780ad000 tx=0x7f6b78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:11.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.674+0000 7f6b6f7fe640 1 -- 172.21.15.7:0/1870106120 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6b7c056120 con 0x7f6b880bd6a0 2024-03-19T01:24:11.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:11.974+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6b88141fa0 con 0x7f6b880bd6a0 2024-03-19T01:24:12.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.918+0000 7f6b6f7fe640 1 -- 172.21.15.7:0/1870106120 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f6b7c055a20 con 0x7f6b880bd6a0 2024-03-19T01:24:12.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b74067660 msgr2=0x7f6b74069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:12.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b74067660 0x7f6b74069b20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b780ad000 tx=0x7f6b78008040 comp rx=0 tx=0).stop 2024-03-19T01:24:12.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880bd6a0 msgr2=0x7f6b8813a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:12.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880bd6a0 0x7f6b8813a070 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00e810 tx=0x7f6b7c00ecd0 comp rx=0 tx=0).stop 2024-03-19T01:24:12.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 shutdown_connections 2024-03-19T01:24:12.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6b74067660 0x7f6b74069b20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:12.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b881410e0 0x7f6b881351b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:12.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b880bd6a0 0x7f6b8813a070 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:12.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 --2- 172.21.15.7:0/1870106120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b880b4600 0x7f6b8814a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:12.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 >> 172.21.15.7:0/1870106120 conn(0x7f6b88072df0 msgr2=0x7f6b8813c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:12.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 shutdown_connections 2024-03-19T01:24:12.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:12.922+0000 7f6b8d2e6640 1 -- 172.21.15.7:0/1870106120 wait complete. 2024-03-19T01:24:12.944 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-19T01:24:12.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-19T01:24:13.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c49507640 1 Processor -- start 2024-03-19T01:24:13.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c49507640 1 -- start start 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c49507640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c440ae070 0x7f4c4414abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c49507640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f4c44141430 0x7f4c441418a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c49507640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c44141de0 con 0x7f4c44074390 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c49507640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f4c4414b130 con 0x7f4c440ae070 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c49507640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f4c440aee60 con 0x7f4c44141430 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c427fc640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c440ae070 0x7f4c4414abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c42ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4c44074390 0x7f4c440ae750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37642/0 (socket says 172.21.15.7:37642) 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c42ffd640 1 -- 172.21.15.7:0/795204439 learned_addr learned my addr 172.21.15.7:0/795204439 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c437fe640 1 --2- 172.21.15.7:0/795204439 >> v2:172.21.15.167:3300/0 conn(0x7f4c44141430 0x7f4c441418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c427fc640 1 -- 172.21.15.7:0/795204439 >> v2:172.21.15.167:3300/0 conn(0x7f4c44141430 msgr2=0x7f4c441418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c427fc640 1 --2- 172.21.15.7:0/795204439 >> v2:172.21.15.167:3300/0 conn(0x7f4c44141430 0x7f4c441418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c427fc640 1 -- 172.21.15.7:0/795204439 >> v1:172.21.15.7:6789/0 conn(0x7f4c44074390 legacy=0x7f4c440ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c427fc640 1 -- 172.21.15.7:0/795204439 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c44074770 con 0x7f4c440ae070 2024-03-19T01:24:13.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c41ffb640 1 -- 172.21.15.7:0/795204439 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2207987213 0 0) 0x7f4c44141de0 con 0x7f4c44074390 2024-03-19T01:24:13.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.070+0000 7f4c437fe640 1 --2- 172.21.15.7:0/795204439 >> v2:172.21.15.167:3300/0 conn(0x7f4c44141430 0x7f4c441418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:13.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c427fc640 1 --2- 172.21.15.7:0/795204439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 0x7f4c4414abf0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4c4406f8c0 tx=0x7f4c3801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7059fee90776a5d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:13.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c41ffb640 1 -- 172.21.15.7:0/795204439 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c380b0070 con 0x7f4c440ae070 2024-03-19T01:24:13.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c41ffb640 1 -- 172.21.15.7:0/795204439 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c380ab600 con 0x7f4c440ae070 2024-03-19T01:24:13.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c41ffb640 1 -- 172.21.15.7:0/795204439 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c380ab9d0 con 0x7f4c440ae070 2024-03-19T01:24:13.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 -- 172.21.15.7:0/795204439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 msgr2=0x7f4c4414abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 --2- 172.21.15.7:0/795204439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 0x7f4c4414abf0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4c4406f8c0 tx=0x7f4c3801c480 comp rx=0 tx=0).stop 2024-03-19T01:24:13.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 -- 172.21.15.7:0/795204439 shutdown_connections 2024-03-19T01:24:13.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 --2- 172.21.15.7:0/795204439 >> v2:172.21.15.167:3300/0 conn(0x7f4c44141430 0x7f4c441418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 --2- 172.21.15.7:0/795204439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 0x7f4c4414abf0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 -- 172.21.15.7:0/795204439 >> 172.21.15.7:0/795204439 conn(0x7f4c4406bc70 msgr2=0x7f4c440b9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:13.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 -- 172.21.15.7:0/795204439 shutdown_connections 2024-03-19T01:24:13.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 -- 172.21.15.7:0/795204439 wait complete. 2024-03-19T01:24:13.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 Processor -- start 2024-03-19T01:24:13.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.074+0000 7f4c49507640 1 -- start start 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c49507640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c44074390 0x7f4c440b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c49507640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 0x7f4c4413bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c49507640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c44141430 0x7f4c441454f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c49507640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4c44154740 con 0x7f4c440ae070 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c49507640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4c440b5630 con 0x7f4c44074390 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c49507640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4c441548c0 con 0x7f4c44141430 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c427fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 0x7f4c4413bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c427fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 0x7f4c4413bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33824/0 (socket says 172.21.15.7:33824) 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c427fc640 1 -- 172.21.15.7:0/3882400725 learned_addr learned my addr 172.21.15.7:0/3882400725 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:13.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c427fc640 1 -- 172.21.15.7:0/3882400725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c44141430 msgr2=0x7f4c441454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c427fc640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c44141430 0x7f4c441454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c427fc640 1 -- 172.21.15.7:0/3882400725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c44074390 msgr2=0x7f4c440b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c427fc640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c44074390 0x7f4c440b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c427fc640 1 -- 172.21.15.7:0/3882400725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c44145a30 con 0x7f4c440ae070 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c427fc640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 0x7f4c4413bea0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4c380ae040 tx=0x7f4c3801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c2b7fe640 1 -- 172.21.15.7:0/3882400725 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c380bc070 con 0x7f4c440ae070 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c2b7fe640 1 -- 172.21.15.7:0/3882400725 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c380b1040 con 0x7f4c440ae070 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c44147a80 con 0x7f4c440ae070 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c2b7fe640 1 -- 172.21.15.7:0/3882400725 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c380b0050 con 0x7f4c440ae070 2024-03-19T01:24:13.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c440b3550 con 0x7f4c440ae070 2024-03-19T01:24:13.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c440b3260 con 0x7f4c440ae070 2024-03-19T01:24:13.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c2b7fe640 1 -- 172.21.15.7:0/3882400725 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4c380bf070 con 0x7f4c440ae070 2024-03-19T01:24:13.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c2b7fe640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4c30067690 0x7f4c30069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c2b7fe640 1 -- 172.21.15.7:0/3882400725 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f4c3811fa00 con 0x7f4c440ae070 2024-03-19T01:24:13.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c42ffd640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4c30067690 0x7f4c30069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.078+0000 7f4c42ffd640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4c30067690 0x7f4c30069b50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c001570 tx=0x7f4c3c007cb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:13.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.082+0000 7f4c2b7fe640 1 -- 172.21.15.7:0/3882400725 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4c380c6020 con 0x7f4c440ae070 2024-03-19T01:24:13.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.278+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f4c440b38e0 con 0x7f4c440ae070 2024-03-19T01:24:13.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.922+0000 7f4c2b7fe640 1 -- 172.21.15.7:0/3882400725 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f4c38016070 con 0x7f4c440ae070 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4c30067690 msgr2=0x7f4c30069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4c30067690 0x7f4c30069b50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c001570 tx=0x7f4c3c007cb0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 msgr2=0x7f4c4413bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 0x7f4c4413bea0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4c380ae040 tx=0x7f4c3801acc0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 shutdown_connections 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4c30067690 0x7f4c30069b50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c44141430 0x7f4c441454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c440ae070 0x7f4c4413bea0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 --2- 172.21.15.7:0/3882400725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c44074390 0x7f4c440b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 >> 172.21.15.7:0/3882400725 conn(0x7f4c4406bc70 msgr2=0x7f4c440b7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:13.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 shutdown_connections 2024-03-19T01:24:13.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:13.926+0000 7f4c49507640 1 -- 172.21.15.7:0/3882400725 wait complete. 2024-03-19T01:24:13.952 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-19T01:24:13.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:24:14.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa252144640 1 Processor -- start 2024-03-19T01:24:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa252144640 1 -- start start 2024-03-19T01:24:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa252144640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa24c09f060 0x7fa24c0af380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa252144640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fa24c06bee0 0x7fa24c147c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa252144640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa24c0aeef0 con 0x7fa24c148170 2024-03-19T01:24:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa252144640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fa24c06ba50 con 0x7fa24c09f060 2024-03-19T01:24:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa252144640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fa24c06de80 con 0x7fa24c06bee0 2024-03-19T01:24:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa24b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa24c09f060 0x7fa24c0af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa24b7fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa24c09f060 0x7fa24c0af380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33830/0 (socket says 172.21.15.7:33830) 2024-03-19T01:24:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa24b7fe640 1 -- 172.21.15.7:0/1074916889 learned_addr learned my addr 172.21.15.7:0/1074916889 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa24affd640 1 --2- 172.21.15.7:0/1074916889 >> v2:172.21.15.167:3300/0 conn(0x7fa24c06bee0 0x7fa24c147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa24b7fe640 1 -- 172.21.15.7:0/1074916889 >> v2:172.21.15.167:3300/0 conn(0x7fa24c06bee0 msgr2=0x7fa24c147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa24b7fe640 1 --2- 172.21.15.7:0/1074916889 >> v2:172.21.15.167:3300/0 conn(0x7fa24c06bee0 0x7fa24c147c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa24b7fe640 1 -- 172.21.15.7:0/1074916889 >> v1:172.21.15.167:6789/0 conn(0x7fa24c148170 legacy=0x7fa24c0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa24b7fe640 1 -- 172.21.15.7:0/1074916889 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24c09f440 con 0x7fa24c09f060 2024-03-19T01:24:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.070+0000 7fa24affd640 1 --2- 172.21.15.7:0/1074916889 >> v2:172.21.15.167:3300/0 conn(0x7fa24c06bee0 0x7fa24c147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24b7fe640 1 --2- 172.21.15.7:0/1074916889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 0x7fa24c0af380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0b0bb0 tx=0x7fa23c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ef200b6887df46 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24a7fc640 1 -- 172.21.15.7:0/1074916889 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23c0b0070 con 0x7fa24c09f060 2024-03-19T01:24:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24a7fc640 1 -- 172.21.15.7:0/1074916889 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa23c0ab600 con 0x7fa24c09f060 2024-03-19T01:24:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24a7fc640 1 -- 172.21.15.7:0/1074916889 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23c0ab9d0 con 0x7fa24c09f060 2024-03-19T01:24:14.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- 172.21.15.7:0/1074916889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 msgr2=0x7fa24c0af380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 --2- 172.21.15.7:0/1074916889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 0x7fa24c0af380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0b0bb0 tx=0x7fa23c01c490 comp rx=0 tx=0).stop 2024-03-19T01:24:14.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- 172.21.15.7:0/1074916889 shutdown_connections 2024-03-19T01:24:14.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 --2- 172.21.15.7:0/1074916889 >> v2:172.21.15.167:3300/0 conn(0x7fa24c06bee0 0x7fa24c147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 --2- 172.21.15.7:0/1074916889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 0x7fa24c0af380 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- 172.21.15.7:0/1074916889 >> 172.21.15.7:0/1074916889 conn(0x7fa24c0b6510 msgr2=0x7fa24c0b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:14.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- 172.21.15.7:0/1074916889 shutdown_connections 2024-03-19T01:24:14.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- 172.21.15.7:0/1074916889 wait complete. 2024-03-19T01:24:14.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 Processor -- start 2024-03-19T01:24:14.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- start start 2024-03-19T01:24:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24c06bee0 0x7fa24c14c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 0x7fa24c14ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa24c148170 0x7fa24c14cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c06ec00 con 0x7fa24c09f060 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c148580 con 0x7fa24c06bee0 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa24c06ed80 con 0x7fa24c148170 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 0x7fa24c14ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 0x7fa24c14ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33834/0 (socket says 172.21.15.7:33834) 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24affd640 1 -- 172.21.15.7:0/3055565539 learned_addr learned my addr 172.21.15.7:0/3055565539 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24b7fe640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24c06bee0 0x7fa24c14c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24bfff640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa24c148170 0x7fa24c14cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24affd640 1 -- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa24c148170 msgr2=0x7fa24c14cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24affd640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa24c148170 0x7fa24c14cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24affd640 1 -- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24c06bee0 msgr2=0x7fa24c14c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24affd640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24c06bee0 0x7fa24c14c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24affd640 1 -- 172.21.15.7:0/3055565539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24c14d640 con 0x7fa24c09f060 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24b7fe640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24c06bee0 0x7fa24c14c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24bfff640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa24c148170 0x7fa24c14cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa24affd640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 0x7fa24c14ca20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa234009fc0 tx=0x7fa23400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa243fff640 1 -- 172.21.15.7:0/3055565539 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23401a070 con 0x7fa24c09f060 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa243fff640 1 -- 172.21.15.7:0/3055565539 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa234003ba0 con 0x7fa24c09f060 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa24c14bba0 con 0x7fa24c09f060 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa243fff640 1 -- 172.21.15.7:0/3055565539 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234004030 con 0x7fa24c09f060 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.074+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa24c148820 con 0x7fa24c09f060 2024-03-19T01:24:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.078+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa24c075460 con 0x7fa24c09f060 2024-03-19T01:24:14.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.078+0000 7fa243fff640 1 -- 172.21.15.7:0/3055565539 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa23401d070 con 0x7fa24c09f060 2024-03-19T01:24:14.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.078+0000 7fa243fff640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa22c067660 0x7fa22c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.078+0000 7fa243fff640 1 -- 172.21.15.7:0/3055565539 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fa234085aa0 con 0x7fa24c09f060 2024-03-19T01:24:14.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.078+0000 7fa24b7fe640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa22c067660 0x7fa22c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.078+0000 7fa24b7fe640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa22c067660 0x7fa22c069b20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa23c0ad000 tx=0x7fa23c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:14.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.082+0000 7fa243fff640 1 -- 172.21.15.7:0/3055565539 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa234010070 con 0x7fa24c09f060 2024-03-19T01:24:14.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa24c146d20 con 0x7fa24c09f060 2024-03-19T01:24:14.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa243fff640 1 -- 172.21.15.7:0/3055565539 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3531 (secure 0 0 0) 0x7fa234056160 con 0x7fa24c09f060 2024-03-19T01:24:14.280 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:14.280 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4753,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/3744754940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3744754940},{"type":"v1","addr":"172.21.15.167:6833","nonce":3744754940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:24:10.251824+0000","modified":"2024-03-19T01:24:13.280725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5001":{"gid":5001,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/4260183338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4260183338},{"type":"v1","addr":"172.21.15.7:6835","nonce":4260183338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5001,"qdb_cluster":[5001]},"id":2}]} 2024-03-19T01:24:14.281 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 16 2024-03-19T01:24:14.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa22c067660 msgr2=0x7fa22c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa22c067660 0x7fa22c069b20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa23c0ad000 tx=0x7fa23c008040 comp rx=0 tx=0).stop 2024-03-19T01:24:14.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 msgr2=0x7fa24c14ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 0x7fa24c14ca20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa234009fc0 tx=0x7fa23400d560 comp rx=0 tx=0).stop 2024-03-19T01:24:14.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 shutdown_connections 2024-03-19T01:24:14.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fa22c067660 0x7fa22c069b20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa24c148170 0x7fa24c14cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c09f060 0x7fa24c14ca20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 --2- 172.21.15.7:0/3055565539 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24c06bee0 0x7fa24c14c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 >> 172.21.15.7:0/3055565539 conn(0x7fa24c0b6510 msgr2=0x7fa24c0b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:14.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 shutdown_connections 2024-03-19T01:24:14.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.278+0000 7fa252144640 1 -- 172.21.15.7:0/3055565539 wait complete. 2024-03-19T01:24:14.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:14.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.418+0000 7f878a45b640 1 Processor -- start 2024-03-19T01:24:14.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f878a45b640 1 -- start start 2024-03-19T01:24:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f878a45b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f878413efd0 0x7f878413f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f878a45b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8784073380 con 0x7f8784072fa0 2024-03-19T01:24:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f878a45b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8784145430 con 0x7f8784145f10 2024-03-19T01:24:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f878a45b640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f8784145710 con 0x7f878413efd0 2024-03-19T01:24:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8783fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8784072fa0 0x7f87840b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37654/0 (socket says 172.21.15.7:37654) 2024-03-19T01:24:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8783fff640 1 -- 172.21.15.7:0/1765061441 learned_addr learned my addr 172.21.15.7:0/1765061441 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f87889f8640 1 --2- 172.21.15.7:0/1765061441 >> v2:172.21.15.167:3300/0 conn(0x7f878413efd0 0x7f878413f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375269729 0 0) 0x7f8784073380 con 0x7f8784072fa0 2024-03-19T01:24:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f876c003530 con 0x7f8784072fa0 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3971282725 0 0) 0x7f876c003530 con 0x7f8784072fa0 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 >> v2:172.21.15.167:3300/0 conn(0x7f878413efd0 msgr2=0x7f878413f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 --2- 172.21.15.7:0/1765061441 >> v2:172.21.15.167:3300/0 conn(0x7f878413efd0 0x7f878413f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 >> v1:172.21.15.167:6789/0 conn(0x7f8784145f10 legacy=0x7f8784071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87841459f0 con 0x7f8784072fa0 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f877c001a00 con 0x7f8784072fa0 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 >> v1:172.21.15.7:6789/0 conn(0x7f8784072fa0 legacy=0x7f87840b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f876c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f876c004810 0x7f876c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f876c0052f0 0x7f876c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8784145710 con 0x7f876c0039c0 2024-03-19T01:24:14.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f876c00c1c0 con 0x7f876c0052f0 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f876c00c3c0 con 0x7f876c004810 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8783fff640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f876c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f87889f8640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f876c0052f0 0x7f876c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f87837fe640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f876c004810 0x7f876c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8783fff640 1 -- 172.21.15.7:0/1765061441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f876c004810 msgr2=0x7f876c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8783fff640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f876c004810 0x7f876c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8783fff640 1 -- 172.21.15.7:0/1765061441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f876c0052f0 msgr2=0x7f876c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8783fff640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f876c0052f0 0x7f876c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f8783fff640 1 -- 172.21.15.7:0/1765061441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c003530 con 0x7f876c0039c0 2024-03-19T01:24:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.422+0000 7f87889f8640 1 -- 172.21.15.7:0/1765061441 reap_dead start 2024-03-19T01:24:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f8783fff640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f876c003de0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f877c00a810 tx=0x7f877c01d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbacc360e7ee8edc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c00a460 con 0x7f876c0039c0 2024-03-19T01:24:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877c0ad650 con 0x7f876c0039c0 2024-03-19T01:24:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f8782ffd640 1 -- 172.21.15.7:0/1765061441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c0ada40 con 0x7f876c0039c0 2024-03-19T01:24:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f878a45b640 1 -- 172.21.15.7:0/1765061441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 msgr2=0x7f876c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f878a45b640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f876c003de0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f877c00a810 tx=0x7f877c01d300 comp rx=0 tx=0).stop 2024-03-19T01:24:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f878a45b640 1 -- 172.21.15.7:0/1765061441 shutdown_connections 2024-03-19T01:24:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f878a45b640 1 --2- 172.21.15.7:0/1765061441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f876c003de0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f878a45b640 1 -- 172.21.15.7:0/1765061441 >> 172.21.15.7:0/1765061441 conn(0x7f878406bc10 msgr2=0x7f87840bac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f878a45b640 1 -- 172.21.15.7:0/1765061441 shutdown_connections 2024-03-19T01:24:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.426+0000 7f878a45b640 1 -- 172.21.15.7:0/1765061441 wait complete. 2024-03-19T01:24:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 Processor -- start 2024-03-19T01:24:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 -- start start 2024-03-19T01:24:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f87841465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8784139290 0x7f8784139710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87841dc350 0x7f87841dc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f87840718e0 con 0x7f876c0039c0 2024-03-19T01:24:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8784071760 con 0x7f8784139290 2024-03-19T01:24:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8784071a60 con 0x7f87841dc350 2024-03-19T01:24:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8783fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f87841465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8783fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f87841465e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33844/0 (socket says 172.21.15.7:33844) 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8783fff640 1 -- 172.21.15.7:0/1045297975 learned_addr learned my addr 172.21.15.7:0/1045297975 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f87837fe640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8784139290 0x7f8784139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f87889f8640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87841dc350 0x7f87841dc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8783fff640 1 -- 172.21.15.7:0/1045297975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87841dc350 msgr2=0x7f87841dc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8783fff640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87841dc350 0x7f87841dc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8783fff640 1 -- 172.21.15.7:0/1045297975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8784139290 msgr2=0x7f8784139710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8783fff640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8784139290 0x7f8784139710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8783fff640 1 -- 172.21.15.7:0/1045297975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8784142290 con 0x7f876c0039c0 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f87837fe640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8784139290 0x7f8784139710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8783fff640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f87841465e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f877c098ac0 tx=0x7f876c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8780ff9640 1 -- 172.21.15.7:0/1045297975 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c001fc0 con 0x7f876c0039c0 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8780ff9640 1 -- 172.21.15.7:0/1045297975 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877c001800 con 0x7f876c0039c0 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8780ff9640 1 -- 172.21.15.7:0/1045297975 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c0019d0 con 0x7f876c0039c0 2024-03-19T01:24:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8784147d60 con 0x7f876c0039c0 2024-03-19T01:24:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8784147300 con 0x7f876c0039c0 2024-03-19T01:24:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8784075460 con 0x7f876c0039c0 2024-03-19T01:24:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8780ff9640 1 -- 172.21.15.7:0/1045297975 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f877c0bc020 con 0x7f876c0039c0 2024-03-19T01:24:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8780ff9640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8770067690 0x7f8770069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f8780ff9640 1 -- 172.21.15.7:0/1045297975 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f877c120400 con 0x7f876c0039c0 2024-03-19T01:24:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.430+0000 7f87837fe640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8770067690 0x7f8770069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.434+0000 7f87837fe640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8770067690 0x7f8770069b50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f87841e02e0 tx=0x7f8778009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:14.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.434+0000 7f8780ff9640 1 -- 172.21.15.7:0/1045297975 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f877c0c3020 con 0x7f876c0039c0 2024-03-19T01:24:14.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.594+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8784073380 con 0x7f876c0039c0 2024-03-19T01:24:14.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.594+0000 7f8780ff9640 1 -- 172.21.15.7:0/1045297975 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16633 (secure 0 0 0) 0x7f877c017010 con 0x7f876c0039c0 2024-03-19T01:24:14.599 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:14.599 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":30,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:24:11.876754+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:24:08.795930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-19T01:24:09.233323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:14.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8770067690 msgr2=0x7f8770069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8770067690 0x7f8770069b50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f87841e02e0 tx=0x7f8778009040 comp rx=0 tx=0).stop 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 msgr2=0x7f87841465e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f87841465e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f877c098ac0 tx=0x7f876c005840 comp rx=0 tx=0).stop 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 shutdown_connections 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8770067690 0x7f8770069b50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87841dc350 0x7f87841dc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8784139290 0x7f8784139710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 --2- 172.21.15.7:0/1045297975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876c0039c0 0x7f87841465e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 >> 172.21.15.7:0/1045297975 conn(0x7f878406bc10 msgr2=0x7f87840b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 shutdown_connections 2024-03-19T01:24:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.598+0000 7f878a45b640 1 -- 172.21.15.7:0/1045297975 wait complete. 2024-03-19T01:24:14.617 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-19T01:24:14.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:24:14.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7d05868640 1 Processor -- start 2024-03-19T01:24:14.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7d05868640 1 -- start start 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7d05868640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7d000b7a90 0x7f7d000ba0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7d05868640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 0x7f7d00141380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7d05868640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d000b9bd0 con 0x7f7d000b88c0 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7d05868640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f7d000b7e70 con 0x7f7d000b7a90 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7d05868640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7d000b95d0 con 0x7f7d0006e0e0 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfeffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7d000b7a90 0x7f7d000ba0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfeffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7d000b7a90 0x7f7d000ba0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33858/0 (socket says 172.21.15.7:33858) 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfeffd640 1 -- 172.21.15.7:0/3160721899 learned_addr learned my addr 172.21.15.7:0/3160721899 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cff7fe640 1 --2- 172.21.15.7:0/3160721899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 0x7f7d00141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfeffd640 1 -- 172.21.15.7:0/3160721899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 msgr2=0x7f7d00141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:14.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfeffd640 1 --2- 172.21.15.7:0/3160721899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 0x7f7d00141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfeffd640 1 -- 172.21.15.7:0/3160721899 >> v1:172.21.15.167:6789/0 conn(0x7f7d000b88c0 legacy=0x7f7d0006d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfeffd640 1 -- 172.21.15.7:0/3160721899 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d000b98b0 con 0x7f7d000b7a90 2024-03-19T01:24:14.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfeffd640 1 --2- 172.21.15.7:0/3160721899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 0x7f7d000ba0b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009870 tx=0x7f7cf001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25c3b33c0f883c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:14.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfdffb640 1 -- 172.21.15.7:0/3160721899 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf00aa620 con 0x7f7d000b7a90 2024-03-19T01:24:14.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfdffb640 1 -- 172.21.15.7:0/3160721899 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf00af070 con 0x7f7d000b7a90 2024-03-19T01:24:14.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.730+0000 7f7cfdffb640 1 -- 172.21.15.7:0/3160721899 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf00aa9e0 con 0x7f7d000b7a90 2024-03-19T01:24:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- 172.21.15.7:0/3160721899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 msgr2=0x7f7d000ba0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 --2- 172.21.15.7:0/3160721899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 0x7f7d000ba0b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009870 tx=0x7f7cf001c4a0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- 172.21.15.7:0/3160721899 shutdown_connections 2024-03-19T01:24:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 --2- 172.21.15.7:0/3160721899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 0x7f7d00141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 --2- 172.21.15.7:0/3160721899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 0x7f7d000ba0b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- 172.21.15.7:0/3160721899 >> 172.21.15.7:0/3160721899 conn(0x7f7d0006b490 msgr2=0x7f7d0006b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- 172.21.15.7:0/3160721899 shutdown_connections 2024-03-19T01:24:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- 172.21.15.7:0/3160721899 wait complete. 2024-03-19T01:24:14.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 Processor -- start 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- start start 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 0x7f7d001e3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 0x7f7d0013b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d000b88c0 0x7f7d00147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7d00154920 con 0x7f7d000b7a90 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001547a0 con 0x7f7d000b88c0 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7d00154aa0 con 0x7f7d0006e0e0 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfe7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 0x7f7d0013b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfe7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 0x7f7d0013b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33872/0 (socket says 172.21.15.7:33872) 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfe7fc640 1 -- 172.21.15.7:0/1231700026 learned_addr learned my addr 172.21.15.7:0/1231700026 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cff7fe640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d000b88c0 0x7f7d00147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfeffd640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 0x7f7d001e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfe7fc640 1 -- 172.21.15.7:0/1231700026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 msgr2=0x7f7d001e3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfe7fc640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 0x7f7d001e3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfe7fc640 1 -- 172.21.15.7:0/1231700026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d000b88c0 msgr2=0x7f7d00147a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfe7fc640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d000b88c0 0x7f7d00147a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfe7fc640 1 -- 172.21.15.7:0/1231700026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d00146130 con 0x7f7d000b7a90 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cff7fe640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d000b88c0 0x7f7d00147a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7cfe7fc640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 0x7f7d0013b210 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7cf400c3a0 tx=0x7f7cf400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7ce7fff640 1 -- 172.21.15.7:0/1231700026 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf40052d0 con 0x7f7d000b7a90 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7ce7fff640 1 -- 172.21.15.7:0/1231700026 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf4005470 con 0x7f7d000b7a90 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7ce7fff640 1 -- 172.21.15.7:0/1231700026 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4007260 con 0x7f7d000b7a90 2024-03-19T01:24:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d001466a0 con 0x7f7d000b7a90 2024-03-19T01:24:14.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.734+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d00145530 con 0x7f7d000b7a90 2024-03-19T01:24:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.738+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d00074130 con 0x7f7d000b7a90 2024-03-19T01:24:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.738+0000 7f7ce7fff640 1 -- 172.21.15.7:0/1231700026 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7cf4007400 con 0x7f7d000b7a90 2024-03-19T01:24:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.738+0000 7f7ce7fff640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7cec067660 0x7f7cec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.738+0000 7f7ce7fff640 1 -- 172.21.15.7:0/1231700026 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f7cf4084d40 con 0x7f7d000b7a90 2024-03-19T01:24:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.738+0000 7f7cfeffd640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7cec067660 0x7f7cec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.738+0000 7f7cfeffd640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7cec067660 0x7f7cec069b20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00ae000 tx=0x7f7cf00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:14.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.746+0000 7f7ce7fff640 1 -- 172.21.15.7:0/1231700026 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7cf40559c0 con 0x7f7d000b7a90 2024-03-19T01:24:14.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.906+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d00075460 con 0x7f7cec067660 2024-03-19T01:24:14.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.922+0000 7f7ce7fff640 1 -- 172.21.15.7:0/1231700026 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251953 (secure 0 0 0) 0x7f7d00075460 con 0x7f7cec067660 2024-03-19T01:24:14.926 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:14.927 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7cec067660 msgr2=0x7f7cec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7cec067660 0x7f7cec069b20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00ae000 tx=0x7f7cf00a7000 comp rx=0 tx=0).stop 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 msgr2=0x7f7d0013b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 0x7f7d0013b210 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7cf400c3a0 tx=0x7f7cf400c860 comp rx=0 tx=0).stop 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 shutdown_connections 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7cec067660 0x7f7cec069b20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d000b88c0 0x7f7d00147a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d000b7a90 0x7f7d0013b210 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 --2- 172.21.15.7:0/1231700026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0006e0e0 0x7f7d001e3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 >> 172.21.15.7:0/1231700026 conn(0x7f7d0006b490 msgr2=0x7f7d00071360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 shutdown_connections 2024-03-19T01:24:14.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:14.926+0000 7f7d05868640 1 -- 172.21.15.7:0/1231700026 wait complete. 2024-03-19T01:24:14.946 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-03-19T01:24:14.902107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236672,"kb_used_data":3848,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738048,"statfs":{"total":773094113280,"available":772851761152,"internally_reserved":0,"allocated":3940352,"data_stored":2215020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001301"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257405+0000","last_change":"2024-03-19T01:24:10.925616+0000","last_active":"2024-03-19T01:24:12.257405+0000","last_peered":"2024-03-19T01:24:12.257405+0000","last_clean":"2024-03-19T01:24:12.257405+0000","last_became_active":"2024-03-19T01:24:10.916249+0000","last_became_peered":"2024-03-19T01:24:10.916249+0000","last_unstale":"2024-03-19T01:24:12.257405+0000","last_undegraded":"2024-03-19T01:24:12.257405+0000","last_fullsized":"2024-03-19T01:24:12.257405+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:24.703434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256773+0000","last_change":"2024-03-19T01:24:10.890929+0000","last_active":"2024-03-19T01:24:12.256773+0000","last_peered":"2024-03-19T01:24:12.256773+0000","last_clean":"2024-03-19T01:24:12.256773+0000","last_became_active":"2024-03-19T01:24:10.890656+0000","last_became_peered":"2024-03-19T01:24:10.890656+0000","last_unstale":"2024-03-19T01:24:12.256773+0000","last_undegraded":"2024-03-19T01:24:12.256773+0000","last_fullsized":"2024-03-19T01:24:12.256773+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:02.824816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259524+0000","last_change":"2024-03-19T01:24:10.918444+0000","last_active":"2024-03-19T01:24:12.259524+0000","last_peered":"2024-03-19T01:24:12.259524+0000","last_clean":"2024-03-19T01:24:12.259524+0000","last_became_active":"2024-03-19T01:24:10.917736+0000","last_became_peered":"2024-03-19T01:24:10.917736+0000","last_unstale":"2024-03-19T01:24:12.259524+0000","last_undegraded":"2024-03-19T01:24:12.259524+0000","last_fullsized":"2024-03-19T01:24:12.259524+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:25.741224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254095+0000","last_change":"2024-03-19T01:24:10.920696+0000","last_active":"2024-03-19T01:24:12.254095+0000","last_peered":"2024-03-19T01:24:12.254095+0000","last_clean":"2024-03-19T01:24:12.254095+0000","last_became_active":"2024-03-19T01:24:10.920447+0000","last_became_peered":"2024-03-19T01:24:10.920447+0000","last_unstale":"2024-03-19T01:24:12.254095+0000","last_undegraded":"2024-03-19T01:24:12.254095+0000","last_fullsized":"2024-03-19T01:24:12.254095+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:34:23.810402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255346+0000","last_change":"2024-03-19T01:24:10.896381+0000","last_active":"2024-03-19T01:24:12.255346+0000","last_peered":"2024-03-19T01:24:12.255346+0000","last_clean":"2024-03-19T01:24:12.255346+0000","last_became_active":"2024-03-19T01:24:10.896068+0000","last_became_peered":"2024-03-19T01:24:10.896068+0000","last_unstale":"2024-03-19T01:24:12.255346+0000","last_undegraded":"2024-03-19T01:24:12.255346+0000","last_fullsized":"2024-03-19T01:24:12.255346+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:09:04.471683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256225+0000","last_change":"2024-03-19T01:24:10.915754+0000","last_active":"2024-03-19T01:24:12.256225+0000","last_peered":"2024-03-19T01:24:12.256225+0000","last_clean":"2024-03-19T01:24:12.256225+0000","last_became_active":"2024-03-19T01:24:10.915435+0000","last_became_peered":"2024-03-19T01:24:10.915435+0000","last_unstale":"2024-03-19T01:24:12.256225+0000","last_undegraded":"2024-03-19T01:24:12.256225+0000","last_fullsized":"2024-03-19T01:24:12.256225+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:10.997227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259410+0000","last_change":"2024-03-19T01:24:10.941882+0000","last_active":"2024-03-19T01:24:12.259410+0000","last_peered":"2024-03-19T01:24:12.259410+0000","last_clean":"2024-03-19T01:24:12.259410+0000","last_became_active":"2024-03-19T01:24:10.940946+0000","last_became_peered":"2024-03-19T01:24:10.940946+0000","last_unstale":"2024-03-19T01:24:12.259410+0000","last_undegraded":"2024-03-19T01:24:12.259410+0000","last_fullsized":"2024-03-19T01:24:12.259410+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:25.741224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256269+0000","last_change":"2024-03-19T01:24:10.909907+0000","last_active":"2024-03-19T01:24:12.256269+0000","last_peered":"2024-03-19T01:24:12.256269+0000","last_clean":"2024-03-19T01:24:12.256269+0000","last_became_active":"2024-03-19T01:24:10.907467+0000","last_became_peered":"2024-03-19T01:24:10.907467+0000","last_unstale":"2024-03-19T01:24:12.256269+0000","last_undegraded":"2024-03-19T01:24:12.256269+0000","last_fullsized":"2024-03-19T01:24:12.256269+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:31.268081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899787+0000","last_change":"2024-03-19T01:24:10.893668+0000","last_active":"2024-03-19T01:24:11.899787+0000","last_peered":"2024-03-19T01:24:11.899787+0000","last_clean":"2024-03-19T01:24:11.899787+0000","last_became_active":"2024-03-19T01:24:10.892777+0000","last_became_peered":"2024-03-19T01:24:10.892777+0000","last_unstale":"2024-03-19T01:24:11.899787+0000","last_undegraded":"2024-03-19T01:24:11.899787+0000","last_fullsized":"2024-03-19T01:24:11.899787+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:34:23.810402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254632+0000","last_change":"2024-03-19T01:24:10.896709+0000","last_active":"2024-03-19T01:24:12.254632+0000","last_peered":"2024-03-19T01:24:12.254632+0000","last_clean":"2024-03-19T01:24:12.254632+0000","last_became_active":"2024-03-19T01:24:10.896441+0000","last_became_peered":"2024-03-19T01:24:10.896441+0000","last_unstale":"2024-03-19T01:24:12.254632+0000","last_undegraded":"2024-03-19T01:24:12.254632+0000","last_fullsized":"2024-03-19T01:24:12.254632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.220412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258814+0000","last_change":"2024-03-19T01:24:10.927268+0000","last_active":"2024-03-19T01:24:12.258814+0000","last_peered":"2024-03-19T01:24:12.258814+0000","last_clean":"2024-03-19T01:24:12.258814+0000","last_became_active":"2024-03-19T01:24:10.926587+0000","last_became_peered":"2024-03-19T01:24:10.926587+0000","last_unstale":"2024-03-19T01:24:12.258814+0000","last_undegraded":"2024-03-19T01:24:12.258814+0000","last_fullsized":"2024-03-19T01:24:12.258814+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:29.353233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258214+0000","last_change":"2024-03-19T01:24:10.920752+0000","last_active":"2024-03-19T01:24:12.258214+0000","last_peered":"2024-03-19T01:24:12.258214+0000","last_clean":"2024-03-19T01:24:12.258214+0000","last_became_active":"2024-03-19T01:24:10.919922+0000","last_became_peered":"2024-03-19T01:24:10.919922+0000","last_unstale":"2024-03-19T01:24:12.258214+0000","last_undegraded":"2024-03-19T01:24:12.258214+0000","last_fullsized":"2024-03-19T01:24:12.258214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:10.997227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255562+0000","last_change":"2024-03-19T01:24:10.888249+0000","last_active":"2024-03-19T01:24:12.255562+0000","last_peered":"2024-03-19T01:24:12.255562+0000","last_clean":"2024-03-19T01:24:12.255562+0000","last_became_active":"2024-03-19T01:24:10.887825+0000","last_became_peered":"2024-03-19T01:24:10.887825+0000","last_unstale":"2024-03-19T01:24:12.255562+0000","last_undegraded":"2024-03-19T01:24:12.255562+0000","last_fullsized":"2024-03-19T01:24:12.255562+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:45.483679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256293+0000","last_change":"2024-03-19T01:24:10.895535+0000","last_active":"2024-03-19T01:24:12.256293+0000","last_peered":"2024-03-19T01:24:12.256293+0000","last_clean":"2024-03-19T01:24:12.256293+0000","last_became_active":"2024-03-19T01:24:10.892774+0000","last_became_peered":"2024-03-19T01:24:10.892774+0000","last_unstale":"2024-03-19T01:24:12.256293+0000","last_undegraded":"2024-03-19T01:24:12.256293+0000","last_fullsized":"2024-03-19T01:24:12.256293+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:25.852042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256441+0000","last_change":"2024-03-19T01:24:10.929611+0000","last_active":"2024-03-19T01:24:12.256441+0000","last_peered":"2024-03-19T01:24:12.256441+0000","last_clean":"2024-03-19T01:24:12.256441+0000","last_became_active":"2024-03-19T01:24:10.928874+0000","last_became_peered":"2024-03-19T01:24:10.928874+0000","last_unstale":"2024-03-19T01:24:12.256441+0000","last_undegraded":"2024-03-19T01:24:12.256441+0000","last_fullsized":"2024-03-19T01:24:12.256441+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:21.075158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257055+0000","last_change":"2024-03-19T01:24:10.925119+0000","last_active":"2024-03-19T01:24:12.257055+0000","last_peered":"2024-03-19T01:24:12.257055+0000","last_clean":"2024-03-19T01:24:12.257055+0000","last_became_active":"2024-03-19T01:24:10.924318+0000","last_became_peered":"2024-03-19T01:24:10.924318+0000","last_unstale":"2024-03-19T01:24:12.257055+0000","last_undegraded":"2024-03-19T01:24:12.257055+0000","last_fullsized":"2024-03-19T01:24:12.257055+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:01.313626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258645+0000","last_change":"2024-03-19T01:24:10.941607+0000","last_active":"2024-03-19T01:24:12.258645+0000","last_peered":"2024-03-19T01:24:12.258645+0000","last_clean":"2024-03-19T01:24:12.258645+0000","last_became_active":"2024-03-19T01:24:10.941086+0000","last_became_peered":"2024-03-19T01:24:10.941086+0000","last_unstale":"2024-03-19T01:24:12.258645+0000","last_undegraded":"2024-03-19T01:24:12.258645+0000","last_fullsized":"2024-03-19T01:24:12.258645+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:31.133305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256827+0000","last_change":"2024-03-19T01:24:10.900794+0000","last_active":"2024-03-19T01:24:12.256827+0000","last_peered":"2024-03-19T01:24:12.256827+0000","last_clean":"2024-03-19T01:24:12.256827+0000","last_became_active":"2024-03-19T01:24:10.900491+0000","last_became_peered":"2024-03-19T01:24:10.900491+0000","last_unstale":"2024-03-19T01:24:12.256827+0000","last_undegraded":"2024-03-19T01:24:12.256827+0000","last_fullsized":"2024-03-19T01:24:12.256827+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:25.741224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257362+0000","last_change":"2024-03-19T01:24:10.900460+0000","last_active":"2024-03-19T01:24:12.257362+0000","last_peered":"2024-03-19T01:24:12.257362+0000","last_clean":"2024-03-19T01:24:12.257362+0000","last_became_active":"2024-03-19T01:24:10.896132+0000","last_became_peered":"2024-03-19T01:24:10.896132+0000","last_unstale":"2024-03-19T01:24:12.257362+0000","last_undegraded":"2024-03-19T01:24:12.257362+0000","last_fullsized":"2024-03-19T01:24:12.257362+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:34:54.783894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255325+0000","last_change":"2024-03-19T01:24:10.923578+0000","last_active":"2024-03-19T01:24:12.255325+0000","last_peered":"2024-03-19T01:24:12.255325+0000","last_clean":"2024-03-19T01:24:12.255325+0000","last_became_active":"2024-03-19T01:24:10.923350+0000","last_became_peered":"2024-03-19T01:24:10.923350+0000","last_unstale":"2024-03-19T01:24:12.255325+0000","last_undegraded":"2024-03-19T01:24:12.255325+0000","last_fullsized":"2024-03-19T01:24:12.255325+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:46.836518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898771+0000","last_change":"2024-03-19T01:24:10.901372+0000","last_active":"2024-03-19T01:24:11.898771+0000","last_peered":"2024-03-19T01:24:11.898771+0000","last_clean":"2024-03-19T01:24:11.898771+0000","last_became_active":"2024-03-19T01:24:10.899361+0000","last_became_peered":"2024-03-19T01:24:10.899361+0000","last_unstale":"2024-03-19T01:24:11.898771+0000","last_undegraded":"2024-03-19T01:24:11.898771+0000","last_fullsized":"2024-03-19T01:24:11.898771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:21.718398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898732+0000","last_change":"2024-03-19T01:24:10.901747+0000","last_active":"2024-03-19T01:24:11.898732+0000","last_peered":"2024-03-19T01:24:11.898732+0000","last_clean":"2024-03-19T01:24:11.898732+0000","last_became_active":"2024-03-19T01:24:10.899147+0000","last_became_peered":"2024-03-19T01:24:10.899147+0000","last_unstale":"2024-03-19T01:24:11.898732+0000","last_undegraded":"2024-03-19T01:24:11.898732+0000","last_fullsized":"2024-03-19T01:24:11.898732+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:16.440853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893089+0000","last_change":"2024-03-19T01:24:10.898522+0000","last_active":"2024-03-19T01:24:11.893089+0000","last_peered":"2024-03-19T01:24:11.893089+0000","last_clean":"2024-03-19T01:24:11.893089+0000","last_became_active":"2024-03-19T01:24:10.898132+0000","last_became_peered":"2024-03-19T01:24:10.898132+0000","last_unstale":"2024-03-19T01:24:11.893089+0000","last_undegraded":"2024-03-19T01:24:11.893089+0000","last_fullsized":"2024-03-19T01:24:11.893089+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:10.992323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256615+0000","last_change":"2024-03-19T01:24:10.895949+0000","last_active":"2024-03-19T01:24:12.256615+0000","last_peered":"2024-03-19T01:24:12.256615+0000","last_clean":"2024-03-19T01:24:12.256615+0000","last_became_active":"2024-03-19T01:24:10.895086+0000","last_became_peered":"2024-03-19T01:24:10.895086+0000","last_unstale":"2024-03-19T01:24:12.256615+0000","last_undegraded":"2024-03-19T01:24:12.256615+0000","last_fullsized":"2024-03-19T01:24:12.256615+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:01.313626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255173+0000","last_change":"2024-03-19T01:24:10.900353+0000","last_active":"2024-03-19T01:24:12.255173+0000","last_peered":"2024-03-19T01:24:12.255173+0000","last_clean":"2024-03-19T01:24:12.255173+0000","last_became_active":"2024-03-19T01:24:10.899858+0000","last_became_peered":"2024-03-19T01:24:10.899858+0000","last_unstale":"2024-03-19T01:24:12.255173+0000","last_undegraded":"2024-03-19T01:24:12.255173+0000","last_fullsized":"2024-03-19T01:24:12.255173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:21.075158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256870+0000","last_change":"2024-03-19T01:24:10.912394+0000","last_active":"2024-03-19T01:24:12.256870+0000","last_peered":"2024-03-19T01:24:12.256870+0000","last_clean":"2024-03-19T01:24:12.256870+0000","last_became_active":"2024-03-19T01:24:10.912126+0000","last_became_peered":"2024-03-19T01:24:10.912126+0000","last_unstale":"2024-03-19T01:24:12.256870+0000","last_undegraded":"2024-03-19T01:24:12.256870+0000","last_fullsized":"2024-03-19T01:24:12.256870+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:04.973168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254915+0000","last_change":"2024-03-19T01:24:10.912084+0000","last_active":"2024-03-19T01:24:12.254915+0000","last_peered":"2024-03-19T01:24:12.254915+0000","last_clean":"2024-03-19T01:24:12.254915+0000","last_became_active":"2024-03-19T01:24:10.911799+0000","last_became_peered":"2024-03-19T01:24:10.911799+0000","last_unstale":"2024-03-19T01:24:12.254915+0000","last_undegraded":"2024-03-19T01:24:12.254915+0000","last_fullsized":"2024-03-19T01:24:12.254915+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:41.581399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258580+0000","last_change":"2024-03-19T01:24:10.918892+0000","last_active":"2024-03-19T01:24:12.258580+0000","last_peered":"2024-03-19T01:24:12.258580+0000","last_clean":"2024-03-19T01:24:12.258580+0000","last_became_active":"2024-03-19T01:24:10.908742+0000","last_became_peered":"2024-03-19T01:24:10.908742+0000","last_unstale":"2024-03-19T01:24:12.258580+0000","last_undegraded":"2024-03-19T01:24:12.258580+0000","last_fullsized":"2024-03-19T01:24:12.258580+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:25.852042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898600+0000","last_change":"2024-03-19T01:24:10.899871+0000","last_active":"2024-03-19T01:24:11.898600+0000","last_peered":"2024-03-19T01:24:11.898600+0000","last_clean":"2024-03-19T01:24:11.898600+0000","last_became_active":"2024-03-19T01:24:10.899078+0000","last_became_peered":"2024-03-19T01:24:10.899078+0000","last_unstale":"2024-03-19T01:24:11.898600+0000","last_undegraded":"2024-03-19T01:24:11.898600+0000","last_fullsized":"2024-03-19T01:24:11.898600+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:04:07.866818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257561+0000","last_change":"2024-03-19T01:24:10.936758+0000","last_active":"2024-03-19T01:24:12.257561+0000","last_peered":"2024-03-19T01:24:12.257561+0000","last_clean":"2024-03-19T01:24:12.257561+0000","last_became_active":"2024-03-19T01:24:10.936147+0000","last_became_peered":"2024-03-19T01:24:10.936147+0000","last_unstale":"2024-03-19T01:24:12.257561+0000","last_undegraded":"2024-03-19T01:24:12.257561+0000","last_fullsized":"2024-03-19T01:24:12.257561+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:18.884118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257309+0000","last_change":"2024-03-19T01:24:10.926690+0000","last_active":"2024-03-19T01:24:12.257309+0000","last_peered":"2024-03-19T01:24:12.257309+0000","last_clean":"2024-03-19T01:24:12.257309+0000","last_became_active":"2024-03-19T01:24:10.916045+0000","last_became_peered":"2024-03-19T01:24:10.916045+0000","last_unstale":"2024-03-19T01:24:12.257309+0000","last_undegraded":"2024-03-19T01:24:12.257309+0000","last_fullsized":"2024-03-19T01:24:12.257309+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:42.809873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259009+0000","last_change":"2024-03-19T01:24:10.933942+0000","last_active":"2024-03-19T01:24:12.259009+0000","last_peered":"2024-03-19T01:24:12.259009+0000","last_clean":"2024-03-19T01:24:12.259009+0000","last_became_active":"2024-03-19T01:24:10.933251+0000","last_became_peered":"2024-03-19T01:24:10.933251+0000","last_unstale":"2024-03-19T01:24:12.259009+0000","last_undegraded":"2024-03-19T01:24:12.259009+0000","last_fullsized":"2024-03-19T01:24:12.259009+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:02.824816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893682+0000","last_change":"2024-03-19T01:24:10.902901+0000","last_active":"2024-03-19T01:24:11.893682+0000","last_peered":"2024-03-19T01:24:11.893682+0000","last_clean":"2024-03-19T01:24:11.893682+0000","last_became_active":"2024-03-19T01:24:10.902616+0000","last_became_peered":"2024-03-19T01:24:10.902616+0000","last_unstale":"2024-03-19T01:24:11.893682+0000","last_undegraded":"2024-03-19T01:24:11.893682+0000","last_fullsized":"2024-03-19T01:24:11.893682+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:52.231516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"29'4","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257935+0000","last_change":"2024-03-19T01:24:09.916318+0000","last_active":"2024-03-19T01:24:12.257935+0000","last_peered":"2024-03-19T01:24:12.257935+0000","last_clean":"2024-03-19T01:24:12.257935+0000","last_became_active":"2024-03-19T01:24:09.915951+0000","last_became_peered":"2024-03-19T01:24:09.915951+0000","last_unstale":"2024-03-19T01:24:12.257935+0000","last_undegraded":"2024-03-19T01:24:12.257935+0000","last_fullsized":"2024-03-19T01:24:12.257935+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:27.207204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257795+0000","last_change":"2024-03-19T01:24:10.931435+0000","last_active":"2024-03-19T01:24:12.257795+0000","last_peered":"2024-03-19T01:24:12.257795+0000","last_clean":"2024-03-19T01:24:12.257795+0000","last_became_active":"2024-03-19T01:24:10.930718+0000","last_became_peered":"2024-03-19T01:24:10.930718+0000","last_unstale":"2024-03-19T01:24:12.257795+0000","last_undegraded":"2024-03-19T01:24:12.257795+0000","last_fullsized":"2024-03-19T01:24:12.257795+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:06.758906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893572+0000","last_change":"2024-03-19T01:24:09.903668+0000","last_active":"2024-03-19T01:24:11.893572+0000","last_peered":"2024-03-19T01:24:11.893572+0000","last_clean":"2024-03-19T01:24:11.893572+0000","last_became_active":"2024-03-19T01:24:09.903243+0000","last_became_peered":"2024-03-19T01:24:09.903243+0000","last_unstale":"2024-03-19T01:24:11.893572+0000","last_undegraded":"2024-03-19T01:24:11.893572+0000","last_fullsized":"2024-03-19T01:24:11.893572+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:09:03.459117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899557+0000","last_change":"2024-03-19T01:24:10.909772+0000","last_active":"2024-03-19T01:24:11.899557+0000","last_peered":"2024-03-19T01:24:11.899557+0000","last_clean":"2024-03-19T01:24:11.899557+0000","last_became_active":"2024-03-19T01:24:10.900892+0000","last_became_peered":"2024-03-19T01:24:10.900892+0000","last_unstale":"2024-03-19T01:24:11.899557+0000","last_undegraded":"2024-03-19T01:24:11.899557+0000","last_fullsized":"2024-03-19T01:24:11.899557+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:06.571691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256530+0000","last_change":"2024-03-19T01:24:09.896081+0000","last_active":"2024-03-19T01:24:12.256530+0000","last_peered":"2024-03-19T01:24:12.256530+0000","last_clean":"2024-03-19T01:24:12.256530+0000","last_became_active":"2024-03-19T01:24:09.894958+0000","last_became_peered":"2024-03-19T01:24:09.894958+0000","last_unstale":"2024-03-19T01:24:12.256530+0000","last_undegraded":"2024-03-19T01:24:12.256530+0000","last_fullsized":"2024-03-19T01:24:12.256530+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:51.381180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255058+0000","last_change":"2024-03-19T01:24:10.900665+0000","last_active":"2024-03-19T01:24:12.255058+0000","last_peered":"2024-03-19T01:24:12.255058+0000","last_clean":"2024-03-19T01:24:12.255058+0000","last_became_active":"2024-03-19T01:24:10.899936+0000","last_became_peered":"2024-03-19T01:24:10.899936+0000","last_unstale":"2024-03-19T01:24:12.255058+0000","last_undegraded":"2024-03-19T01:24:12.255058+0000","last_fullsized":"2024-03-19T01:24:12.255058+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:31.268081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255400+0000","last_change":"2024-03-19T01:24:09.903637+0000","last_active":"2024-03-19T01:24:12.255400+0000","last_peered":"2024-03-19T01:24:12.255400+0000","last_clean":"2024-03-19T01:24:12.255400+0000","last_became_active":"2024-03-19T01:24:09.902710+0000","last_became_peered":"2024-03-19T01:24:09.902710+0000","last_unstale":"2024-03-19T01:24:12.255400+0000","last_undegraded":"2024-03-19T01:24:12.255400+0000","last_fullsized":"2024-03-19T01:24:12.255400+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:58.982015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257969+0000","last_change":"2024-03-19T01:24:10.933028+0000","last_active":"2024-03-19T01:24:12.257969+0000","last_peered":"2024-03-19T01:24:12.257969+0000","last_clean":"2024-03-19T01:24:12.257969+0000","last_became_active":"2024-03-19T01:24:10.930379+0000","last_became_peered":"2024-03-19T01:24:10.930379+0000","last_unstale":"2024-03-19T01:24:12.257969+0000","last_undegraded":"2024-03-19T01:24:12.257969+0000","last_fullsized":"2024-03-19T01:24:12.257969+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:10.997227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898934+0000","last_change":"2024-03-19T01:24:09.905155+0000","last_active":"2024-03-19T01:24:11.898934+0000","last_peered":"2024-03-19T01:24:11.898934+0000","last_clean":"2024-03-19T01:24:11.898934+0000","last_became_active":"2024-03-19T01:24:09.904789+0000","last_became_peered":"2024-03-19T01:24:09.904789+0000","last_unstale":"2024-03-19T01:24:11.898934+0000","last_undegraded":"2024-03-19T01:24:11.898934+0000","last_fullsized":"2024-03-19T01:24:11.898934+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:44.471112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257219+0000","last_change":"2024-03-19T01:24:10.907507+0000","last_active":"2024-03-19T01:24:12.257219+0000","last_peered":"2024-03-19T01:24:12.257219+0000","last_clean":"2024-03-19T01:24:12.257219+0000","last_became_active":"2024-03-19T01:24:10.906586+0000","last_became_peered":"2024-03-19T01:24:10.906586+0000","last_unstale":"2024-03-19T01:24:12.257219+0000","last_undegraded":"2024-03-19T01:24:12.257219+0000","last_fullsized":"2024-03-19T01:24:12.257219+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:18.884118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898584+0000","last_change":"2024-03-19T01:24:09.905939+0000","last_active":"2024-03-19T01:24:11.898584+0000","last_peered":"2024-03-19T01:24:11.898584+0000","last_clean":"2024-03-19T01:24:11.898584+0000","last_became_active":"2024-03-19T01:24:09.905075+0000","last_became_peered":"2024-03-19T01:24:09.905075+0000","last_unstale":"2024-03-19T01:24:11.898584+0000","last_undegraded":"2024-03-19T01:24:11.898584+0000","last_fullsized":"2024-03-19T01:24:11.898584+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:09:03.459117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898782+0000","last_change":"2024-03-19T01:24:10.910309+0000","last_active":"2024-03-19T01:24:11.898782+0000","last_peered":"2024-03-19T01:24:11.898782+0000","last_clean":"2024-03-19T01:24:11.898782+0000","last_became_active":"2024-03-19T01:24:10.904133+0000","last_became_peered":"2024-03-19T01:24:10.904133+0000","last_unstale":"2024-03-19T01:24:11.898782+0000","last_undegraded":"2024-03-19T01:24:11.898782+0000","last_fullsized":"2024-03-19T01:24:11.898782+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:50.681266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256408+0000","last_change":"2024-03-19T01:24:09.904601+0000","last_active":"2024-03-19T01:24:12.256408+0000","last_peered":"2024-03-19T01:24:12.256408+0000","last_clean":"2024-03-19T01:24:12.256408+0000","last_became_active":"2024-03-19T01:24:09.903840+0000","last_became_peered":"2024-03-19T01:24:09.903840+0000","last_unstale":"2024-03-19T01:24:12.256408+0000","last_undegraded":"2024-03-19T01:24:12.256408+0000","last_fullsized":"2024-03-19T01:24:12.256408+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:51.218949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.253752+0000","last_change":"2024-03-19T01:24:10.919208+0000","last_active":"2024-03-19T01:24:12.253752+0000","last_peered":"2024-03-19T01:24:12.253752+0000","last_clean":"2024-03-19T01:24:12.253752+0000","last_became_active":"2024-03-19T01:24:10.918773+0000","last_became_peered":"2024-03-19T01:24:10.918773+0000","last_unstale":"2024-03-19T01:24:12.253752+0000","last_undegraded":"2024-03-19T01:24:12.253752+0000","last_fullsized":"2024-03-19T01:24:12.253752+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:21.718398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898215+0000","last_change":"2024-03-19T01:24:09.909924+0000","last_active":"2024-03-19T01:24:11.898215+0000","last_peered":"2024-03-19T01:24:11.898215+0000","last_clean":"2024-03-19T01:24:11.898215+0000","last_became_active":"2024-03-19T01:24:09.909565+0000","last_became_peered":"2024-03-19T01:24:09.909565+0000","last_unstale":"2024-03-19T01:24:11.898215+0000","last_undegraded":"2024-03-19T01:24:11.898215+0000","last_fullsized":"2024-03-19T01:24:11.898215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:33.389593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.260090+0000","last_change":"2024-03-19T01:24:10.916418+0000","last_active":"2024-03-19T01:24:12.260090+0000","last_peered":"2024-03-19T01:24:12.260090+0000","last_clean":"2024-03-19T01:24:12.260090+0000","last_became_active":"2024-03-19T01:24:10.915682+0000","last_became_peered":"2024-03-19T01:24:10.915682+0000","last_unstale":"2024-03-19T01:24:12.260090+0000","last_undegraded":"2024-03-19T01:24:12.260090+0000","last_fullsized":"2024-03-19T01:24:12.260090+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:04.973168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256780+0000","last_change":"2024-03-19T01:24:09.918125+0000","last_active":"2024-03-19T01:24:12.256780+0000","last_peered":"2024-03-19T01:24:12.256780+0000","last_clean":"2024-03-19T01:24:12.256780+0000","last_became_active":"2024-03-19T01:24:09.917740+0000","last_became_peered":"2024-03-19T01:24:09.917740+0000","last_unstale":"2024-03-19T01:24:12.256780+0000","last_undegraded":"2024-03-19T01:24:12.256780+0000","last_fullsized":"2024-03-19T01:24:12.256780+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:15.428287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256721+0000","last_change":"2024-03-19T01:24:10.900847+0000","last_active":"2024-03-19T01:24:12.256721+0000","last_peered":"2024-03-19T01:24:12.256721+0000","last_clean":"2024-03-19T01:24:12.256721+0000","last_became_active":"2024-03-19T01:24:10.896110+0000","last_became_peered":"2024-03-19T01:24:10.896110+0000","last_unstale":"2024-03-19T01:24:12.256721+0000","last_undegraded":"2024-03-19T01:24:12.256721+0000","last_fullsized":"2024-03-19T01:24:12.256721+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:29.353233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893871+0000","last_change":"2024-03-19T01:24:09.901817+0000","last_active":"2024-03-19T01:24:11.893871+0000","last_peered":"2024-03-19T01:24:11.893871+0000","last_clean":"2024-03-19T01:24:11.893871+0000","last_became_active":"2024-03-19T01:24:09.899759+0000","last_became_peered":"2024-03-19T01:24:09.899759+0000","last_unstale":"2024-03-19T01:24:11.893871+0000","last_undegraded":"2024-03-19T01:24:11.893871+0000","last_fullsized":"2024-03-19T01:24:11.893871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:57.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256372+0000","last_change":"2024-03-19T01:24:10.901428+0000","last_active":"2024-03-19T01:24:12.256372+0000","last_peered":"2024-03-19T01:24:12.256372+0000","last_clean":"2024-03-19T01:24:12.256372+0000","last_became_active":"2024-03-19T01:24:10.901135+0000","last_became_peered":"2024-03-19T01:24:10.901135+0000","last_unstale":"2024-03-19T01:24:12.256372+0000","last_undegraded":"2024-03-19T01:24:12.256372+0000","last_fullsized":"2024-03-19T01:24:12.256372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:31.133305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"29'3","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259604+0000","last_change":"2024-03-19T01:24:09.910513+0000","last_active":"2024-03-19T01:24:12.259604+0000","last_peered":"2024-03-19T01:24:12.259604+0000","last_clean":"2024-03-19T01:24:12.259604+0000","last_became_active":"2024-03-19T01:24:09.909742+0000","last_became_peered":"2024-03-19T01:24:09.909742+0000","last_unstale":"2024-03-19T01:24:12.259604+0000","last_undegraded":"2024-03-19T01:24:12.259604+0000","last_fullsized":"2024-03-19T01:24:12.259604+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:51.381180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259266+0000","last_change":"2024-03-19T01:24:10.920680+0000","last_active":"2024-03-19T01:24:12.259266+0000","last_peered":"2024-03-19T01:24:12.259266+0000","last_clean":"2024-03-19T01:24:12.259266+0000","last_became_active":"2024-03-19T01:24:10.919131+0000","last_became_peered":"2024-03-19T01:24:10.919131+0000","last_unstale":"2024-03-19T01:24:12.259266+0000","last_undegraded":"2024-03-19T01:24:12.259266+0000","last_fullsized":"2024-03-19T01:24:12.259266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:31.133305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"29'6","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256530+0000","last_change":"2024-03-19T01:24:09.897489+0000","last_active":"2024-03-19T01:24:12.256530+0000","last_peered":"2024-03-19T01:24:12.256530+0000","last_clean":"2024-03-19T01:24:12.256530+0000","last_became_active":"2024-03-19T01:24:09.897087+0000","last_became_peered":"2024-03-19T01:24:09.897087+0000","last_unstale":"2024-03-19T01:24:12.256530+0000","last_undegraded":"2024-03-19T01:24:12.256530+0000","last_fullsized":"2024-03-19T01:24:12.256530+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:20.705832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.253762+0000","last_change":"2024-03-19T01:24:10.919116+0000","last_active":"2024-03-19T01:24:12.253762+0000","last_peered":"2024-03-19T01:24:12.253762+0000","last_clean":"2024-03-19T01:24:12.253762+0000","last_became_active":"2024-03-19T01:24:10.918614+0000","last_became_peered":"2024-03-19T01:24:10.918614+0000","last_unstale":"2024-03-19T01:24:12.253762+0000","last_undegraded":"2024-03-19T01:24:12.253762+0000","last_fullsized":"2024-03-19T01:24:12.253762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:50.681266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899001+0000","last_change":"2024-03-19T01:24:09.907779+0000","last_active":"2024-03-19T01:24:11.899001+0000","last_peered":"2024-03-19T01:24:11.899001+0000","last_clean":"2024-03-19T01:24:11.899001+0000","last_became_active":"2024-03-19T01:24:09.906981+0000","last_became_peered":"2024-03-19T01:24:09.906981+0000","last_unstale":"2024-03-19T01:24:11.899001+0000","last_undegraded":"2024-03-19T01:24:11.899001+0000","last_fullsized":"2024-03-19T01:24:11.899001+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:57.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257119+0000","last_change":"2024-03-19T01:24:10.892791+0000","last_active":"2024-03-19T01:24:12.257119+0000","last_peered":"2024-03-19T01:24:12.257119+0000","last_clean":"2024-03-19T01:24:12.257119+0000","last_became_active":"2024-03-19T01:24:10.892506+0000","last_became_peered":"2024-03-19T01:24:10.892506+0000","last_unstale":"2024-03-19T01:24:12.257119+0000","last_undegraded":"2024-03-19T01:24:12.257119+0000","last_fullsized":"2024-03-19T01:24:12.257119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:44:51.202538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.253920+0000","last_change":"2024-03-19T01:24:09.913132+0000","last_active":"2024-03-19T01:24:12.253920+0000","last_peered":"2024-03-19T01:24:12.253920+0000","last_clean":"2024-03-19T01:24:12.253920+0000","last_became_active":"2024-03-19T01:24:09.912821+0000","last_became_peered":"2024-03-19T01:24:09.912821+0000","last_unstale":"2024-03-19T01:24:12.253920+0000","last_undegraded":"2024-03-19T01:24:12.253920+0000","last_fullsized":"2024-03-19T01:24:12.253920+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:23.690868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256656+0000","last_change":"2024-03-19T01:24:10.935675+0000","last_active":"2024-03-19T01:24:12.256656+0000","last_peered":"2024-03-19T01:24:12.256656+0000","last_clean":"2024-03-19T01:24:12.256656+0000","last_became_active":"2024-03-19T01:24:10.934014+0000","last_became_peered":"2024-03-19T01:24:10.934014+0000","last_unstale":"2024-03-19T01:24:12.256656+0000","last_undegraded":"2024-03-19T01:24:12.256656+0000","last_fullsized":"2024-03-19T01:24:12.256656+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:21.075158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256115+0000","last_change":"2024-03-19T01:24:09.916657+0000","last_active":"2024-03-19T01:24:12.256115+0000","last_peered":"2024-03-19T01:24:12.256115+0000","last_clean":"2024-03-19T01:24:12.256115+0000","last_became_active":"2024-03-19T01:24:09.916205+0000","last_became_peered":"2024-03-19T01:24:09.916205+0000","last_unstale":"2024-03-19T01:24:12.256115+0000","last_undegraded":"2024-03-19T01:24:12.256115+0000","last_fullsized":"2024-03-19T01:24:12.256115+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:09.979757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257709+0000","last_change":"2024-03-19T01:24:10.930261+0000","last_active":"2024-03-19T01:24:12.257709+0000","last_peered":"2024-03-19T01:24:12.257709+0000","last_clean":"2024-03-19T01:24:12.257709+0000","last_became_active":"2024-03-19T01:24:10.929567+0000","last_became_peered":"2024-03-19T01:24:10.929567+0000","last_unstale":"2024-03-19T01:24:12.257709+0000","last_undegraded":"2024-03-19T01:24:12.257709+0000","last_fullsized":"2024-03-19T01:24:12.257709+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:51:22.972765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"29'8","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256120+0000","last_change":"2024-03-19T01:24:09.897217+0000","last_active":"2024-03-19T01:24:12.256120+0000","last_peered":"2024-03-19T01:24:12.256120+0000","last_clean":"2024-03-19T01:24:12.256120+0000","last_became_active":"2024-03-19T01:24:09.896713+0000","last_became_peered":"2024-03-19T01:24:09.896713+0000","last_unstale":"2024-03-19T01:24:12.256120+0000","last_undegraded":"2024-03-19T01:24:12.256120+0000","last_fullsized":"2024-03-19T01:24:12.256120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:15.961961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899956+0000","last_change":"2024-03-19T01:24:10.921859+0000","last_active":"2024-03-19T01:24:11.899956+0000","last_peered":"2024-03-19T01:24:11.899956+0000","last_clean":"2024-03-19T01:24:11.899956+0000","last_became_active":"2024-03-19T01:24:10.921012+0000","last_became_peered":"2024-03-19T01:24:10.921012+0000","last_unstale":"2024-03-19T01:24:11.899956+0000","last_undegraded":"2024-03-19T01:24:11.899956+0000","last_fullsized":"2024-03-19T01:24:11.899956+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.220412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255054+0000","last_change":"2024-03-19T01:24:09.900799+0000","last_active":"2024-03-19T01:24:12.255054+0000","last_peered":"2024-03-19T01:24:12.255054+0000","last_clean":"2024-03-19T01:24:12.255054+0000","last_became_active":"2024-03-19T01:24:09.899109+0000","last_became_peered":"2024-03-19T01:24:09.899109+0000","last_unstale":"2024-03-19T01:24:12.255054+0000","last_undegraded":"2024-03-19T01:24:12.255054+0000","last_fullsized":"2024-03-19T01:24:12.255054+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:37.893827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259311+0000","last_change":"2024-03-19T01:24:10.930718+0000","last_active":"2024-03-19T01:24:12.259311+0000","last_peered":"2024-03-19T01:24:12.259311+0000","last_clean":"2024-03-19T01:24:12.259311+0000","last_became_active":"2024-03-19T01:24:10.930059+0000","last_became_peered":"2024-03-19T01:24:10.930059+0000","last_unstale":"2024-03-19T01:24:12.259311+0000","last_undegraded":"2024-03-19T01:24:12.259311+0000","last_fullsized":"2024-03-19T01:24:12.259311+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:29.353233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899369+0000","last_change":"2024-03-19T01:24:09.907889+0000","last_active":"2024-03-19T01:24:11.899369+0000","last_peered":"2024-03-19T01:24:11.899369+0000","last_clean":"2024-03-19T01:24:11.899369+0000","last_became_active":"2024-03-19T01:24:09.907220+0000","last_became_peered":"2024-03-19T01:24:09.907220+0000","last_unstale":"2024-03-19T01:24:11.899369+0000","last_undegraded":"2024-03-19T01:24:11.899369+0000","last_fullsized":"2024-03-19T01:24:11.899369+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:37.893827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254046+0000","last_change":"2024-03-19T01:24:10.912260+0000","last_active":"2024-03-19T01:24:12.254046+0000","last_peered":"2024-03-19T01:24:12.254046+0000","last_clean":"2024-03-19T01:24:12.254046+0000","last_became_active":"2024-03-19T01:24:10.911887+0000","last_became_peered":"2024-03-19T01:24:10.911887+0000","last_unstale":"2024-03-19T01:24:12.254046+0000","last_undegraded":"2024-03-19T01:24:12.254046+0000","last_fullsized":"2024-03-19T01:24:12.254046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:06.571691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899511+0000","last_change":"2024-03-19T01:24:09.898311+0000","last_active":"2024-03-19T01:24:11.899511+0000","last_peered":"2024-03-19T01:24:11.899511+0000","last_clean":"2024-03-19T01:24:11.899511+0000","last_became_active":"2024-03-19T01:24:09.897568+0000","last_became_peered":"2024-03-19T01:24:09.897568+0000","last_unstale":"2024-03-19T01:24:11.899511+0000","last_undegraded":"2024-03-19T01:24:11.899511+0000","last_fullsized":"2024-03-19T01:24:11.899511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:45.823952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255141+0000","last_change":"2024-03-19T01:24:10.917973+0000","last_active":"2024-03-19T01:24:12.255141+0000","last_peered":"2024-03-19T01:24:12.255141+0000","last_clean":"2024-03-19T01:24:12.255141+0000","last_became_active":"2024-03-19T01:24:10.917679+0000","last_became_peered":"2024-03-19T01:24:10.917679+0000","last_unstale":"2024-03-19T01:24:12.255141+0000","last_undegraded":"2024-03-19T01:24:12.255141+0000","last_fullsized":"2024-03-19T01:24:12.255141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:28.219770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899679+0000","last_change":"2024-03-19T01:24:09.908585+0000","last_active":"2024-03-19T01:24:11.899679+0000","last_peered":"2024-03-19T01:24:11.899679+0000","last_clean":"2024-03-19T01:24:11.899679+0000","last_became_active":"2024-03-19T01:24:09.908242+0000","last_became_peered":"2024-03-19T01:24:09.908242+0000","last_unstale":"2024-03-19T01:24:11.899679+0000","last_undegraded":"2024-03-19T01:24:11.899679+0000","last_fullsized":"2024-03-19T01:24:11.899679+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:21.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255082+0000","last_change":"2024-03-19T01:24:10.899196+0000","last_active":"2024-03-19T01:24:12.255082+0000","last_peered":"2024-03-19T01:24:12.255082+0000","last_clean":"2024-03-19T01:24:12.255082+0000","last_became_active":"2024-03-19T01:24:10.898842+0000","last_became_peered":"2024-03-19T01:24:10.898842+0000","last_unstale":"2024-03-19T01:24:12.255082+0000","last_undegraded":"2024-03-19T01:24:12.255082+0000","last_fullsized":"2024-03-19T01:24:12.255082+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:22.112910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.892956+0000","last_change":"2024-03-19T01:24:09.899480+0000","last_active":"2024-03-19T01:24:11.892956+0000","last_peered":"2024-03-19T01:24:11.892956+0000","last_clean":"2024-03-19T01:24:11.892956+0000","last_became_active":"2024-03-19T01:24:09.898875+0000","last_became_peered":"2024-03-19T01:24:09.898875+0000","last_unstale":"2024-03-19T01:24:11.892956+0000","last_undegraded":"2024-03-19T01:24:11.892956+0000","last_fullsized":"2024-03-19T01:24:11.892956+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:37.893827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255799+0000","last_change":"2024-03-19T01:24:10.891031+0000","last_active":"2024-03-19T01:24:12.255799+0000","last_peered":"2024-03-19T01:24:12.255799+0000","last_clean":"2024-03-19T01:24:12.255799+0000","last_became_active":"2024-03-19T01:24:10.890748+0000","last_became_peered":"2024-03-19T01:24:10.890748+0000","last_unstale":"2024-03-19T01:24:12.255799+0000","last_undegraded":"2024-03-19T01:24:12.255799+0000","last_fullsized":"2024-03-19T01:24:12.255799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:58.241465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258008+0000","last_change":"2024-03-19T01:24:09.912771+0000","last_active":"2024-03-19T01:24:12.258008+0000","last_peered":"2024-03-19T01:24:12.258008+0000","last_clean":"2024-03-19T01:24:12.258008+0000","last_became_active":"2024-03-19T01:24:09.912383+0000","last_became_peered":"2024-03-19T01:24:09.912383+0000","last_unstale":"2024-03-19T01:24:12.258008+0000","last_undegraded":"2024-03-19T01:24:12.258008+0000","last_fullsized":"2024-03-19T01:24:12.258008+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:21.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254411+0000","last_change":"2024-03-19T01:24:10.887383+0000","last_active":"2024-03-19T01:24:12.254411+0000","last_peered":"2024-03-19T01:24:12.254411+0000","last_clean":"2024-03-19T01:24:12.254411+0000","last_became_active":"2024-03-19T01:24:10.887141+0000","last_became_peered":"2024-03-19T01:24:10.887141+0000","last_unstale":"2024-03-19T01:24:12.254411+0000","last_undegraded":"2024-03-19T01:24:12.254411+0000","last_fullsized":"2024-03-19T01:24:12.254411+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:34.402159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898215+0000","last_change":"2024-03-19T01:24:09.890802+0000","last_active":"2024-03-19T01:24:11.898215+0000","last_peered":"2024-03-19T01:24:11.898215+0000","last_clean":"2024-03-19T01:24:11.898215+0000","last_became_active":"2024-03-19T01:24:09.889918+0000","last_became_peered":"2024-03-19T01:24:09.889918+0000","last_unstale":"2024-03-19T01:24:11.898215+0000","last_undegraded":"2024-03-19T01:24:11.898215+0000","last_fullsized":"2024-03-19T01:24:11.898215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:58.773246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256194+0000","last_change":"2024-03-19T01:24:10.936768+0000","last_active":"2024-03-19T01:24:12.256194+0000","last_peered":"2024-03-19T01:24:12.256194+0000","last_clean":"2024-03-19T01:24:12.256194+0000","last_became_active":"2024-03-19T01:24:10.936087+0000","last_became_peered":"2024-03-19T01:24:10.936087+0000","last_unstale":"2024-03-19T01:24:12.256194+0000","last_undegraded":"2024-03-19T01:24:12.256194+0000","last_fullsized":"2024-03-19T01:24:12.256194+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:41.581399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256753+0000","last_change":"2024-03-19T01:24:09.918453+0000","last_active":"2024-03-19T01:24:12.256753+0000","last_peered":"2024-03-19T01:24:12.256753+0000","last_clean":"2024-03-19T01:24:12.256753+0000","last_became_active":"2024-03-19T01:24:09.917806+0000","last_became_peered":"2024-03-19T01:24:09.917806+0000","last_unstale":"2024-03-19T01:24:12.256753+0000","last_undegraded":"2024-03-19T01:24:12.256753+0000","last_fullsized":"2024-03-19T01:24:12.256753+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:09.979757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257564+0000","last_change":"2024-03-19T01:24:10.920280+0000","last_active":"2024-03-19T01:24:12.257564+0000","last_peered":"2024-03-19T01:24:12.257564+0000","last_clean":"2024-03-19T01:24:12.257564+0000","last_became_active":"2024-03-19T01:24:10.919985+0000","last_became_peered":"2024-03-19T01:24:10.919985+0000","last_unstale":"2024-03-19T01:24:12.257564+0000","last_undegraded":"2024-03-19T01:24:12.257564+0000","last_fullsized":"2024-03-19T01:24:12.257564+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:38.906393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255184+0000","last_change":"2024-03-19T01:24:09.916657+0000","last_active":"2024-03-19T01:24:12.255184+0000","last_peered":"2024-03-19T01:24:12.255184+0000","last_clean":"2024-03-19T01:24:12.255184+0000","last_became_active":"2024-03-19T01:24:09.916140+0000","last_became_peered":"2024-03-19T01:24:09.916140+0000","last_unstale":"2024-03-19T01:24:12.255184+0000","last_undegraded":"2024-03-19T01:24:12.255184+0000","last_fullsized":"2024-03-19T01:24:12.255184+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:41.797307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259496+0000","last_change":"2024-03-19T01:24:10.915155+0000","last_active":"2024-03-19T01:24:12.259496+0000","last_peered":"2024-03-19T01:24:12.259496+0000","last_clean":"2024-03-19T01:24:12.259496+0000","last_became_active":"2024-03-19T01:24:10.914437+0000","last_became_peered":"2024-03-19T01:24:10.914437+0000","last_unstale":"2024-03-19T01:24:12.259496+0000","last_undegraded":"2024-03-19T01:24:12.259496+0000","last_fullsized":"2024-03-19T01:24:12.259496+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:02.824816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893841+0000","last_change":"2024-03-19T01:24:09.904351+0000","last_active":"2024-03-19T01:24:11.893841+0000","last_peered":"2024-03-19T01:24:11.893841+0000","last_clean":"2024-03-19T01:24:11.893841+0000","last_became_active":"2024-03-19T01:24:09.902472+0000","last_became_peered":"2024-03-19T01:24:09.902472+0000","last_unstale":"2024-03-19T01:24:11.893841+0000","last_undegraded":"2024-03-19T01:24:11.893841+0000","last_fullsized":"2024-03-19T01:24:11.893841+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:44.471112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893060+0000","last_change":"2024-03-19T01:24:10.893855+0000","last_active":"2024-03-19T01:24:11.893060+0000","last_peered":"2024-03-19T01:24:11.893060+0000","last_clean":"2024-03-19T01:24:11.893060+0000","last_became_active":"2024-03-19T01:24:10.893531+0000","last_became_peered":"2024-03-19T01:24:10.893531+0000","last_unstale":"2024-03-19T01:24:11.893060+0000","last_undegraded":"2024-03-19T01:24:11.893060+0000","last_fullsized":"2024-03-19T01:24:11.893060+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:16.974528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254558+0000","last_change":"2024-03-19T01:24:09.901870+0000","last_active":"2024-03-19T01:24:12.254558+0000","last_peered":"2024-03-19T01:24:12.254558+0000","last_clean":"2024-03-19T01:24:12.254558+0000","last_became_active":"2024-03-19T01:24:09.899450+0000","last_became_peered":"2024-03-19T01:24:09.899450+0000","last_unstale":"2024-03-19T01:24:12.254558+0000","last_undegraded":"2024-03-19T01:24:12.254558+0000","last_fullsized":"2024-03-19T01:24:12.254558+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:05.746340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258160+0000","last_change":"2024-03-19T01:24:10.915432+0000","last_active":"2024-03-19T01:24:12.258160+0000","last_peered":"2024-03-19T01:24:12.258160+0000","last_clean":"2024-03-19T01:24:12.258160+0000","last_became_active":"2024-03-19T01:24:10.914630+0000","last_became_peered":"2024-03-19T01:24:10.914630+0000","last_unstale":"2024-03-19T01:24:12.258160+0000","last_undegraded":"2024-03-19T01:24:12.258160+0000","last_fullsized":"2024-03-19T01:24:12.258160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:52:34.148842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893189+0000","last_change":"2024-03-19T01:24:09.916012+0000","last_active":"2024-03-19T01:24:11.893189+0000","last_peered":"2024-03-19T01:24:11.893189+0000","last_clean":"2024-03-19T01:24:11.893189+0000","last_became_active":"2024-03-19T01:24:09.915593+0000","last_became_peered":"2024-03-19T01:24:09.915593+0000","last_unstale":"2024-03-19T01:24:11.893189+0000","last_undegraded":"2024-03-19T01:24:11.893189+0000","last_fullsized":"2024-03-19T01:24:11.893189+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:41.797307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.892983+0000","last_change":"2024-03-19T01:24:10.910324+0000","last_active":"2024-03-19T01:24:11.892983+0000","last_peered":"2024-03-19T01:24:11.892983+0000","last_clean":"2024-03-19T01:24:11.892983+0000","last_became_active":"2024-03-19T01:24:10.910029+0000","last_became_peered":"2024-03-19T01:24:10.910029+0000","last_unstale":"2024-03-19T01:24:11.892983+0000","last_undegraded":"2024-03-19T01:24:11.892983+0000","last_fullsized":"2024-03-19T01:24:11.892983+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:23.319987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.253824+0000","last_change":"2024-03-19T01:24:09.916901+0000","last_active":"2024-03-19T01:24:12.253824+0000","last_peered":"2024-03-19T01:24:12.253824+0000","last_clean":"2024-03-19T01:24:12.253824+0000","last_became_active":"2024-03-19T01:24:09.916155+0000","last_became_peered":"2024-03-19T01:24:09.916155+0000","last_unstale":"2024-03-19T01:24:12.253824+0000","last_undegraded":"2024-03-19T01:24:12.253824+0000","last_fullsized":"2024-03-19T01:24:12.253824+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:51:21.960199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256091+0000","last_change":"2024-03-19T01:24:10.894775+0000","last_active":"2024-03-19T01:24:12.256091+0000","last_peered":"2024-03-19T01:24:12.256091+0000","last_clean":"2024-03-19T01:24:12.256091+0000","last_became_active":"2024-03-19T01:24:10.894372+0000","last_became_peered":"2024-03-19T01:24:10.894372+0000","last_unstale":"2024-03-19T01:24:12.256091+0000","last_undegraded":"2024-03-19T01:24:12.256091+0000","last_fullsized":"2024-03-19T01:24:12.256091+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:18.884118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898621+0000","last_change":"2024-03-19T01:24:09.916961+0000","last_active":"2024-03-19T01:24:11.898621+0000","last_peered":"2024-03-19T01:24:11.898621+0000","last_clean":"2024-03-19T01:24:11.898621+0000","last_became_active":"2024-03-19T01:24:09.916341+0000","last_became_peered":"2024-03-19T01:24:09.916341+0000","last_unstale":"2024-03-19T01:24:11.898621+0000","last_undegraded":"2024-03-19T01:24:11.898621+0000","last_fullsized":"2024-03-19T01:24:11.898621+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:27.207204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898472+0000","last_change":"2024-03-19T01:24:10.902504+0000","last_active":"2024-03-19T01:24:11.898472+0000","last_peered":"2024-03-19T01:24:11.898472+0000","last_clean":"2024-03-19T01:24:11.898472+0000","last_became_active":"2024-03-19T01:24:10.900667+0000","last_became_peered":"2024-03-19T01:24:10.900667+0000","last_unstale":"2024-03-19T01:24:11.898472+0000","last_undegraded":"2024-03-19T01:24:11.898472+0000","last_fullsized":"2024-03-19T01:24:11.898472+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:52.393746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256016+0000","last_change":"2024-03-19T01:24:09.912281+0000","last_active":"2024-03-19T01:24:12.256016+0000","last_peered":"2024-03-19T01:24:12.256016+0000","last_clean":"2024-03-19T01:24:12.256016+0000","last_became_active":"2024-03-19T01:24:09.911745+0000","last_became_peered":"2024-03-19T01:24:09.911745+0000","last_unstale":"2024-03-19T01:24:12.256016+0000","last_undegraded":"2024-03-19T01:24:12.256016+0000","last_fullsized":"2024-03-19T01:24:12.256016+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:22.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258025+0000","last_change":"2024-03-19T01:24:10.925298+0000","last_active":"2024-03-19T01:24:12.258025+0000","last_peered":"2024-03-19T01:24:12.258025+0000","last_clean":"2024-03-19T01:24:12.258025+0000","last_became_active":"2024-03-19T01:24:10.924512+0000","last_became_peered":"2024-03-19T01:24:10.924512+0000","last_unstale":"2024-03-19T01:24:12.258025+0000","last_undegraded":"2024-03-19T01:24:12.258025+0000","last_fullsized":"2024-03-19T01:24:12.258025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:25.852042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256403+0000","last_change":"2024-03-19T01:24:09.912688+0000","last_active":"2024-03-19T01:24:12.256403+0000","last_peered":"2024-03-19T01:24:12.256403+0000","last_clean":"2024-03-19T01:24:12.256403+0000","last_became_active":"2024-03-19T01:24:09.911954+0000","last_became_peered":"2024-03-19T01:24:09.911954+0000","last_unstale":"2024-03-19T01:24:12.256403+0000","last_undegraded":"2024-03-19T01:24:12.256403+0000","last_fullsized":"2024-03-19T01:24:12.256403+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:33.389593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":187,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257190+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:24:12.257190+0000","last_peered":"2024-03-19T01:24:12.257190+0000","last_clean":"2024-03-19T01:24:12.257190+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:24:12.257190+0000","last_undegraded":"2024-03-19T01:24:12.257190+0000","last_fullsized":"2024-03-19T01:24:12.257190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":933888,"data_stored":719141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":344064,"data_stored":129299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":344064,"data_stored":129355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":344064,"data_stored":128787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":130317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":339968,"data_stored":128753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":130317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":929792,"data_stored":719051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:24:14.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:24:15.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.106+0000 7ff8d2b16640 1 Processor -- start 2024-03-19T01:24:15.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.106+0000 7ff8d2b16640 1 -- start start 2024-03-19T01:24:15.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8d2b16640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff8cc0b05f0 0x7ff8cc1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8d2b16640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8cc074770 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8d2b16640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8cc06fe80 con 0x7ff8cc074390 2024-03-19T01:24:15.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8d2b16640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7ff8cc070160 con 0x7ff8cc0b05f0 2024-03-19T01:24:15.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8cbfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff8cc0b05f0 0x7ff8cc1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8cbfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff8cc0b05f0 0x7ff8cc1480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33876/0 (socket says 172.21.15.7:33876) 2024-03-19T01:24:15.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8cbfff640 1 -- 172.21.15.7:0/3180184948 learned_addr learned my addr 172.21.15.7:0/3180184948 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:15.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8cbfff640 1 -- 172.21.15.7:0/3180184948 >> v1:172.21.15.167:6789/0 conn(0x7ff8cc074390 legacy=0x7ff8cc0b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:15.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8cbfff640 1 -- 172.21.15.7:0/3180184948 >> v1:172.21.15.7:6789/0 conn(0x7ff8cc13dfe0 legacy=0x7ff8cc14e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8cbfff640 1 -- 172.21.15.7:0/3180184948 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8cc1475e0 con 0x7ff8cc0b05f0 2024-03-19T01:24:15.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.110+0000 7ff8cbfff640 1 --2- 172.21.15.7:0/3180184948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc0b05f0 0x7ff8cc1480c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009e50 tx=0x7ff8b401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecd63b661689adf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8cb7fe640 1 -- 172.21.15.7:0/3180184948 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b40a5620 con 0x7ff8cc0b05f0 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8cb7fe640 1 -- 172.21.15.7:0/3180184948 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b40a57c0 con 0x7ff8cc0b05f0 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8cb7fe640 1 -- 172.21.15.7:0/3180184948 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b40a5b90 con 0x7ff8cc0b05f0 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 -- 172.21.15.7:0/3180184948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc0b05f0 msgr2=0x7ff8cc1480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 --2- 172.21.15.7:0/3180184948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc0b05f0 0x7ff8cc1480c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009e50 tx=0x7ff8b401c960 comp rx=0 tx=0).stop 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 -- 172.21.15.7:0/3180184948 shutdown_connections 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 --2- 172.21.15.7:0/3180184948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc0b05f0 0x7ff8cc1480c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 -- 172.21.15.7:0/3180184948 >> 172.21.15.7:0/3180184948 conn(0x7ff8cc0b5bf0 msgr2=0x7ff8cc0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 -- 172.21.15.7:0/3180184948 shutdown_connections 2024-03-19T01:24:15.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 -- 172.21.15.7:0/3180184948 wait complete. 2024-03-19T01:24:15.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 Processor -- start 2024-03-19T01:24:15.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 -- start start 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8cc074390 0x7ff8cc143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8cc0b05f0 0x7ff8cc1358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc13dfe0 0x7ff8cc135de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc152c40 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc152ac0 con 0x7ff8cc074390 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d2b16640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff8cc152dc0 con 0x7ff8cc0b05f0 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d10b3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc13dfe0 0x7ff8cc135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d10b3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc13dfe0 0x7ff8cc135de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33884/0 (socket says 172.21.15.7:33884) 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d10b3640 1 -- 172.21.15.7:0/1588500224 learned_addr learned my addr 172.21.15.7:0/1588500224 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8cbfff640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8cc0b05f0 0x7ff8cc1358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d08b2640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8cc074390 0x7ff8cc143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d10b3640 1 -- 172.21.15.7:0/1588500224 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8cc0b05f0 msgr2=0x7ff8cc1358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d10b3640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8cc0b05f0 0x7ff8cc1358a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d10b3640 1 -- 172.21.15.7:0/1588500224 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8cc074390 msgr2=0x7ff8cc143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d10b3640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8cc074390 0x7ff8cc143d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.114+0000 7ff8d10b3640 1 -- 172.21.15.7:0/1588500224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8cc13ebb0 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8d10b3640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc13dfe0 0x7ff8cc135de0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00c3c0 tx=0x7ff8bc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:15.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8c97fa640 1 -- 172.21.15.7:0/1588500224 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc0052d0 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8c97fa640 1 -- 172.21.15.7:0/1588500224 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8bc005470 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8cc1419f0 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8c97fa640 1 -- 172.21.15.7:0/1588500224 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc007240 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8cc1ddaf0 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8cc075460 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8c97fa640 1 -- 172.21.15.7:0/1588500224 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff8bc0073e0 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8c97fa640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8ac067690 0x7ff8ac069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8c97fa640 1 -- 172.21.15.7:0/1588500224 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7ff8bc084c50 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8d08b2640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8ac067690 0x7ff8ac069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.118+0000 7ff8d08b2640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8ac067690 0x7ff8ac069b50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4004e30 tx=0x7ff8c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:15.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.122+0000 7ff8c97fa640 1 -- 172.21.15.7:0/1588500224 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff8bc0558c0 con 0x7ff8cc13dfe0 2024-03-19T01:24:15.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.278+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff8cc134d20 con 0x7ff8ac067690 2024-03-19T01:24:15.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.294+0000 7ff8c97fa640 1 -- 172.21.15.7:0/1588500224 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251953 (secure 0 0 0) 0x7ff8cc134d20 con 0x7ff8ac067690 2024-03-19T01:24:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.298+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8ac067690 msgr2=0x7ff8ac069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.298+0000 7ff8d2b16640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8ac067690 0x7ff8ac069b50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4004e30 tx=0x7ff8c4009040 comp rx=0 tx=0).stop 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.298+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc13dfe0 msgr2=0x7ff8cc135de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.298+0000 7ff8d2b16640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc13dfe0 0x7ff8cc135de0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00c3c0 tx=0x7ff8bc00c880 comp rx=0 tx=0).stop 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.302+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 shutdown_connections 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.302+0000 7ff8d2b16640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff8ac067690 0x7ff8ac069b50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.302+0000 7ff8d2b16640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff8cc0b05f0 0x7ff8cc1358a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.302+0000 7ff8d2b16640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8cc13dfe0 0x7ff8cc135de0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.302+0000 7ff8d2b16640 1 --2- 172.21.15.7:0/1588500224 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8cc074390 0x7ff8cc143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.302+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 >> 172.21.15.7:0/1588500224 conn(0x7ff8cc0b5bf0 msgr2=0x7ff8cc0b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.302+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 shutdown_connections 2024-03-19T01:24:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.302+0000 7ff8d2b16640 1 -- 172.21.15.7:0/1588500224 wait complete. 2024-03-19T01:24:15.319 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-03-19T01:24:14.902107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236672,"kb_used_data":3848,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738048,"statfs":{"total":773094113280,"available":772851761152,"internally_reserved":0,"allocated":3940352,"data_stored":2215020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001301"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257405+0000","last_change":"2024-03-19T01:24:10.925616+0000","last_active":"2024-03-19T01:24:12.257405+0000","last_peered":"2024-03-19T01:24:12.257405+0000","last_clean":"2024-03-19T01:24:12.257405+0000","last_became_active":"2024-03-19T01:24:10.916249+0000","last_became_peered":"2024-03-19T01:24:10.916249+0000","last_unstale":"2024-03-19T01:24:12.257405+0000","last_undegraded":"2024-03-19T01:24:12.257405+0000","last_fullsized":"2024-03-19T01:24:12.257405+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:24.703434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256773+0000","last_change":"2024-03-19T01:24:10.890929+0000","last_active":"2024-03-19T01:24:12.256773+0000","last_peered":"2024-03-19T01:24:12.256773+0000","last_clean":"2024-03-19T01:24:12.256773+0000","last_became_active":"2024-03-19T01:24:10.890656+0000","last_became_peered":"2024-03-19T01:24:10.890656+0000","last_unstale":"2024-03-19T01:24:12.256773+0000","last_undegraded":"2024-03-19T01:24:12.256773+0000","last_fullsized":"2024-03-19T01:24:12.256773+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:02.824816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259524+0000","last_change":"2024-03-19T01:24:10.918444+0000","last_active":"2024-03-19T01:24:12.259524+0000","last_peered":"2024-03-19T01:24:12.259524+0000","last_clean":"2024-03-19T01:24:12.259524+0000","last_became_active":"2024-03-19T01:24:10.917736+0000","last_became_peered":"2024-03-19T01:24:10.917736+0000","last_unstale":"2024-03-19T01:24:12.259524+0000","last_undegraded":"2024-03-19T01:24:12.259524+0000","last_fullsized":"2024-03-19T01:24:12.259524+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:25.741224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254095+0000","last_change":"2024-03-19T01:24:10.920696+0000","last_active":"2024-03-19T01:24:12.254095+0000","last_peered":"2024-03-19T01:24:12.254095+0000","last_clean":"2024-03-19T01:24:12.254095+0000","last_became_active":"2024-03-19T01:24:10.920447+0000","last_became_peered":"2024-03-19T01:24:10.920447+0000","last_unstale":"2024-03-19T01:24:12.254095+0000","last_undegraded":"2024-03-19T01:24:12.254095+0000","last_fullsized":"2024-03-19T01:24:12.254095+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:34:23.810402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255346+0000","last_change":"2024-03-19T01:24:10.896381+0000","last_active":"2024-03-19T01:24:12.255346+0000","last_peered":"2024-03-19T01:24:12.255346+0000","last_clean":"2024-03-19T01:24:12.255346+0000","last_became_active":"2024-03-19T01:24:10.896068+0000","last_became_peered":"2024-03-19T01:24:10.896068+0000","last_unstale":"2024-03-19T01:24:12.255346+0000","last_undegraded":"2024-03-19T01:24:12.255346+0000","last_fullsized":"2024-03-19T01:24:12.255346+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:09:04.471683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256225+0000","last_change":"2024-03-19T01:24:10.915754+0000","last_active":"2024-03-19T01:24:12.256225+0000","last_peered":"2024-03-19T01:24:12.256225+0000","last_clean":"2024-03-19T01:24:12.256225+0000","last_became_active":"2024-03-19T01:24:10.915435+0000","last_became_peered":"2024-03-19T01:24:10.915435+0000","last_unstale":"2024-03-19T01:24:12.256225+0000","last_undegraded":"2024-03-19T01:24:12.256225+0000","last_fullsized":"2024-03-19T01:24:12.256225+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:10.997227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259410+0000","last_change":"2024-03-19T01:24:10.941882+0000","last_active":"2024-03-19T01:24:12.259410+0000","last_peered":"2024-03-19T01:24:12.259410+0000","last_clean":"2024-03-19T01:24:12.259410+0000","last_became_active":"2024-03-19T01:24:10.940946+0000","last_became_peered":"2024-03-19T01:24:10.940946+0000","last_unstale":"2024-03-19T01:24:12.259410+0000","last_undegraded":"2024-03-19T01:24:12.259410+0000","last_fullsized":"2024-03-19T01:24:12.259410+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:25.741224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256269+0000","last_change":"2024-03-19T01:24:10.909907+0000","last_active":"2024-03-19T01:24:12.256269+0000","last_peered":"2024-03-19T01:24:12.256269+0000","last_clean":"2024-03-19T01:24:12.256269+0000","last_became_active":"2024-03-19T01:24:10.907467+0000","last_became_peered":"2024-03-19T01:24:10.907467+0000","last_unstale":"2024-03-19T01:24:12.256269+0000","last_undegraded":"2024-03-19T01:24:12.256269+0000","last_fullsized":"2024-03-19T01:24:12.256269+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:31.268081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899787+0000","last_change":"2024-03-19T01:24:10.893668+0000","last_active":"2024-03-19T01:24:11.899787+0000","last_peered":"2024-03-19T01:24:11.899787+0000","last_clean":"2024-03-19T01:24:11.899787+0000","last_became_active":"2024-03-19T01:24:10.892777+0000","last_became_peered":"2024-03-19T01:24:10.892777+0000","last_unstale":"2024-03-19T01:24:11.899787+0000","last_undegraded":"2024-03-19T01:24:11.899787+0000","last_fullsized":"2024-03-19T01:24:11.899787+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:34:23.810402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254632+0000","last_change":"2024-03-19T01:24:10.896709+0000","last_active":"2024-03-19T01:24:12.254632+0000","last_peered":"2024-03-19T01:24:12.254632+0000","last_clean":"2024-03-19T01:24:12.254632+0000","last_became_active":"2024-03-19T01:24:10.896441+0000","last_became_peered":"2024-03-19T01:24:10.896441+0000","last_unstale":"2024-03-19T01:24:12.254632+0000","last_undegraded":"2024-03-19T01:24:12.254632+0000","last_fullsized":"2024-03-19T01:24:12.254632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.220412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258814+0000","last_change":"2024-03-19T01:24:10.927268+0000","last_active":"2024-03-19T01:24:12.258814+0000","last_peered":"2024-03-19T01:24:12.258814+0000","last_clean":"2024-03-19T01:24:12.258814+0000","last_became_active":"2024-03-19T01:24:10.926587+0000","last_became_peered":"2024-03-19T01:24:10.926587+0000","last_unstale":"2024-03-19T01:24:12.258814+0000","last_undegraded":"2024-03-19T01:24:12.258814+0000","last_fullsized":"2024-03-19T01:24:12.258814+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:29.353233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258214+0000","last_change":"2024-03-19T01:24:10.920752+0000","last_active":"2024-03-19T01:24:12.258214+0000","last_peered":"2024-03-19T01:24:12.258214+0000","last_clean":"2024-03-19T01:24:12.258214+0000","last_became_active":"2024-03-19T01:24:10.919922+0000","last_became_peered":"2024-03-19T01:24:10.919922+0000","last_unstale":"2024-03-19T01:24:12.258214+0000","last_undegraded":"2024-03-19T01:24:12.258214+0000","last_fullsized":"2024-03-19T01:24:12.258214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:10.997227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255562+0000","last_change":"2024-03-19T01:24:10.888249+0000","last_active":"2024-03-19T01:24:12.255562+0000","last_peered":"2024-03-19T01:24:12.255562+0000","last_clean":"2024-03-19T01:24:12.255562+0000","last_became_active":"2024-03-19T01:24:10.887825+0000","last_became_peered":"2024-03-19T01:24:10.887825+0000","last_unstale":"2024-03-19T01:24:12.255562+0000","last_undegraded":"2024-03-19T01:24:12.255562+0000","last_fullsized":"2024-03-19T01:24:12.255562+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:45.483679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256293+0000","last_change":"2024-03-19T01:24:10.895535+0000","last_active":"2024-03-19T01:24:12.256293+0000","last_peered":"2024-03-19T01:24:12.256293+0000","last_clean":"2024-03-19T01:24:12.256293+0000","last_became_active":"2024-03-19T01:24:10.892774+0000","last_became_peered":"2024-03-19T01:24:10.892774+0000","last_unstale":"2024-03-19T01:24:12.256293+0000","last_undegraded":"2024-03-19T01:24:12.256293+0000","last_fullsized":"2024-03-19T01:24:12.256293+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:25.852042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256441+0000","last_change":"2024-03-19T01:24:10.929611+0000","last_active":"2024-03-19T01:24:12.256441+0000","last_peered":"2024-03-19T01:24:12.256441+0000","last_clean":"2024-03-19T01:24:12.256441+0000","last_became_active":"2024-03-19T01:24:10.928874+0000","last_became_peered":"2024-03-19T01:24:10.928874+0000","last_unstale":"2024-03-19T01:24:12.256441+0000","last_undegraded":"2024-03-19T01:24:12.256441+0000","last_fullsized":"2024-03-19T01:24:12.256441+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:21.075158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257055+0000","last_change":"2024-03-19T01:24:10.925119+0000","last_active":"2024-03-19T01:24:12.257055+0000","last_peered":"2024-03-19T01:24:12.257055+0000","last_clean":"2024-03-19T01:24:12.257055+0000","last_became_active":"2024-03-19T01:24:10.924318+0000","last_became_peered":"2024-03-19T01:24:10.924318+0000","last_unstale":"2024-03-19T01:24:12.257055+0000","last_undegraded":"2024-03-19T01:24:12.257055+0000","last_fullsized":"2024-03-19T01:24:12.257055+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:01.313626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258645+0000","last_change":"2024-03-19T01:24:10.941607+0000","last_active":"2024-03-19T01:24:12.258645+0000","last_peered":"2024-03-19T01:24:12.258645+0000","last_clean":"2024-03-19T01:24:12.258645+0000","last_became_active":"2024-03-19T01:24:10.941086+0000","last_became_peered":"2024-03-19T01:24:10.941086+0000","last_unstale":"2024-03-19T01:24:12.258645+0000","last_undegraded":"2024-03-19T01:24:12.258645+0000","last_fullsized":"2024-03-19T01:24:12.258645+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:31.133305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256827+0000","last_change":"2024-03-19T01:24:10.900794+0000","last_active":"2024-03-19T01:24:12.256827+0000","last_peered":"2024-03-19T01:24:12.256827+0000","last_clean":"2024-03-19T01:24:12.256827+0000","last_became_active":"2024-03-19T01:24:10.900491+0000","last_became_peered":"2024-03-19T01:24:10.900491+0000","last_unstale":"2024-03-19T01:24:12.256827+0000","last_undegraded":"2024-03-19T01:24:12.256827+0000","last_fullsized":"2024-03-19T01:24:12.256827+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:21:25.741224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257362+0000","last_change":"2024-03-19T01:24:10.900460+0000","last_active":"2024-03-19T01:24:12.257362+0000","last_peered":"2024-03-19T01:24:12.257362+0000","last_clean":"2024-03-19T01:24:12.257362+0000","last_became_active":"2024-03-19T01:24:10.896132+0000","last_became_peered":"2024-03-19T01:24:10.896132+0000","last_unstale":"2024-03-19T01:24:12.257362+0000","last_undegraded":"2024-03-19T01:24:12.257362+0000","last_fullsized":"2024-03-19T01:24:12.257362+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:34:54.783894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255325+0000","last_change":"2024-03-19T01:24:10.923578+0000","last_active":"2024-03-19T01:24:12.255325+0000","last_peered":"2024-03-19T01:24:12.255325+0000","last_clean":"2024-03-19T01:24:12.255325+0000","last_became_active":"2024-03-19T01:24:10.923350+0000","last_became_peered":"2024-03-19T01:24:10.923350+0000","last_unstale":"2024-03-19T01:24:12.255325+0000","last_undegraded":"2024-03-19T01:24:12.255325+0000","last_fullsized":"2024-03-19T01:24:12.255325+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:46.836518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898771+0000","last_change":"2024-03-19T01:24:10.901372+0000","last_active":"2024-03-19T01:24:11.898771+0000","last_peered":"2024-03-19T01:24:11.898771+0000","last_clean":"2024-03-19T01:24:11.898771+0000","last_became_active":"2024-03-19T01:24:10.899361+0000","last_became_peered":"2024-03-19T01:24:10.899361+0000","last_unstale":"2024-03-19T01:24:11.898771+0000","last_undegraded":"2024-03-19T01:24:11.898771+0000","last_fullsized":"2024-03-19T01:24:11.898771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:21.718398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898732+0000","last_change":"2024-03-19T01:24:10.901747+0000","last_active":"2024-03-19T01:24:11.898732+0000","last_peered":"2024-03-19T01:24:11.898732+0000","last_clean":"2024-03-19T01:24:11.898732+0000","last_became_active":"2024-03-19T01:24:10.899147+0000","last_became_peered":"2024-03-19T01:24:10.899147+0000","last_unstale":"2024-03-19T01:24:11.898732+0000","last_undegraded":"2024-03-19T01:24:11.898732+0000","last_fullsized":"2024-03-19T01:24:11.898732+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:16.440853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893089+0000","last_change":"2024-03-19T01:24:10.898522+0000","last_active":"2024-03-19T01:24:11.893089+0000","last_peered":"2024-03-19T01:24:11.893089+0000","last_clean":"2024-03-19T01:24:11.893089+0000","last_became_active":"2024-03-19T01:24:10.898132+0000","last_became_peered":"2024-03-19T01:24:10.898132+0000","last_unstale":"2024-03-19T01:24:11.893089+0000","last_undegraded":"2024-03-19T01:24:11.893089+0000","last_fullsized":"2024-03-19T01:24:11.893089+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:10.992323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256615+0000","last_change":"2024-03-19T01:24:10.895949+0000","last_active":"2024-03-19T01:24:12.256615+0000","last_peered":"2024-03-19T01:24:12.256615+0000","last_clean":"2024-03-19T01:24:12.256615+0000","last_became_active":"2024-03-19T01:24:10.895086+0000","last_became_peered":"2024-03-19T01:24:10.895086+0000","last_unstale":"2024-03-19T01:24:12.256615+0000","last_undegraded":"2024-03-19T01:24:12.256615+0000","last_fullsized":"2024-03-19T01:24:12.256615+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:01.313626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255173+0000","last_change":"2024-03-19T01:24:10.900353+0000","last_active":"2024-03-19T01:24:12.255173+0000","last_peered":"2024-03-19T01:24:12.255173+0000","last_clean":"2024-03-19T01:24:12.255173+0000","last_became_active":"2024-03-19T01:24:10.899858+0000","last_became_peered":"2024-03-19T01:24:10.899858+0000","last_unstale":"2024-03-19T01:24:12.255173+0000","last_undegraded":"2024-03-19T01:24:12.255173+0000","last_fullsized":"2024-03-19T01:24:12.255173+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:21.075158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256870+0000","last_change":"2024-03-19T01:24:10.912394+0000","last_active":"2024-03-19T01:24:12.256870+0000","last_peered":"2024-03-19T01:24:12.256870+0000","last_clean":"2024-03-19T01:24:12.256870+0000","last_became_active":"2024-03-19T01:24:10.912126+0000","last_became_peered":"2024-03-19T01:24:10.912126+0000","last_unstale":"2024-03-19T01:24:12.256870+0000","last_undegraded":"2024-03-19T01:24:12.256870+0000","last_fullsized":"2024-03-19T01:24:12.256870+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:04.973168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254915+0000","last_change":"2024-03-19T01:24:10.912084+0000","last_active":"2024-03-19T01:24:12.254915+0000","last_peered":"2024-03-19T01:24:12.254915+0000","last_clean":"2024-03-19T01:24:12.254915+0000","last_became_active":"2024-03-19T01:24:10.911799+0000","last_became_peered":"2024-03-19T01:24:10.911799+0000","last_unstale":"2024-03-19T01:24:12.254915+0000","last_undegraded":"2024-03-19T01:24:12.254915+0000","last_fullsized":"2024-03-19T01:24:12.254915+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:41.581399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258580+0000","last_change":"2024-03-19T01:24:10.918892+0000","last_active":"2024-03-19T01:24:12.258580+0000","last_peered":"2024-03-19T01:24:12.258580+0000","last_clean":"2024-03-19T01:24:12.258580+0000","last_became_active":"2024-03-19T01:24:10.908742+0000","last_became_peered":"2024-03-19T01:24:10.908742+0000","last_unstale":"2024-03-19T01:24:12.258580+0000","last_undegraded":"2024-03-19T01:24:12.258580+0000","last_fullsized":"2024-03-19T01:24:12.258580+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:25.852042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898600+0000","last_change":"2024-03-19T01:24:10.899871+0000","last_active":"2024-03-19T01:24:11.898600+0000","last_peered":"2024-03-19T01:24:11.898600+0000","last_clean":"2024-03-19T01:24:11.898600+0000","last_became_active":"2024-03-19T01:24:10.899078+0000","last_became_peered":"2024-03-19T01:24:10.899078+0000","last_unstale":"2024-03-19T01:24:11.898600+0000","last_undegraded":"2024-03-19T01:24:11.898600+0000","last_fullsized":"2024-03-19T01:24:11.898600+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:04:07.866818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257561+0000","last_change":"2024-03-19T01:24:10.936758+0000","last_active":"2024-03-19T01:24:12.257561+0000","last_peered":"2024-03-19T01:24:12.257561+0000","last_clean":"2024-03-19T01:24:12.257561+0000","last_became_active":"2024-03-19T01:24:10.936147+0000","last_became_peered":"2024-03-19T01:24:10.936147+0000","last_unstale":"2024-03-19T01:24:12.257561+0000","last_undegraded":"2024-03-19T01:24:12.257561+0000","last_fullsized":"2024-03-19T01:24:12.257561+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:18.884118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257309+0000","last_change":"2024-03-19T01:24:10.926690+0000","last_active":"2024-03-19T01:24:12.257309+0000","last_peered":"2024-03-19T01:24:12.257309+0000","last_clean":"2024-03-19T01:24:12.257309+0000","last_became_active":"2024-03-19T01:24:10.916045+0000","last_became_peered":"2024-03-19T01:24:10.916045+0000","last_unstale":"2024-03-19T01:24:12.257309+0000","last_undegraded":"2024-03-19T01:24:12.257309+0000","last_fullsized":"2024-03-19T01:24:12.257309+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:42.809873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259009+0000","last_change":"2024-03-19T01:24:10.933942+0000","last_active":"2024-03-19T01:24:12.259009+0000","last_peered":"2024-03-19T01:24:12.259009+0000","last_clean":"2024-03-19T01:24:12.259009+0000","last_became_active":"2024-03-19T01:24:10.933251+0000","last_became_peered":"2024-03-19T01:24:10.933251+0000","last_unstale":"2024-03-19T01:24:12.259009+0000","last_undegraded":"2024-03-19T01:24:12.259009+0000","last_fullsized":"2024-03-19T01:24:12.259009+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:02.824816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893682+0000","last_change":"2024-03-19T01:24:10.902901+0000","last_active":"2024-03-19T01:24:11.893682+0000","last_peered":"2024-03-19T01:24:11.893682+0000","last_clean":"2024-03-19T01:24:11.893682+0000","last_became_active":"2024-03-19T01:24:10.902616+0000","last_became_peered":"2024-03-19T01:24:10.902616+0000","last_unstale":"2024-03-19T01:24:11.893682+0000","last_undegraded":"2024-03-19T01:24:11.893682+0000","last_fullsized":"2024-03-19T01:24:11.893682+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:52.231516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"29'4","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257935+0000","last_change":"2024-03-19T01:24:09.916318+0000","last_active":"2024-03-19T01:24:12.257935+0000","last_peered":"2024-03-19T01:24:12.257935+0000","last_clean":"2024-03-19T01:24:12.257935+0000","last_became_active":"2024-03-19T01:24:09.915951+0000","last_became_peered":"2024-03-19T01:24:09.915951+0000","last_unstale":"2024-03-19T01:24:12.257935+0000","last_undegraded":"2024-03-19T01:24:12.257935+0000","last_fullsized":"2024-03-19T01:24:12.257935+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:27.207204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257795+0000","last_change":"2024-03-19T01:24:10.931435+0000","last_active":"2024-03-19T01:24:12.257795+0000","last_peered":"2024-03-19T01:24:12.257795+0000","last_clean":"2024-03-19T01:24:12.257795+0000","last_became_active":"2024-03-19T01:24:10.930718+0000","last_became_peered":"2024-03-19T01:24:10.930718+0000","last_unstale":"2024-03-19T01:24:12.257795+0000","last_undegraded":"2024-03-19T01:24:12.257795+0000","last_fullsized":"2024-03-19T01:24:12.257795+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:06.758906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893572+0000","last_change":"2024-03-19T01:24:09.903668+0000","last_active":"2024-03-19T01:24:11.893572+0000","last_peered":"2024-03-19T01:24:11.893572+0000","last_clean":"2024-03-19T01:24:11.893572+0000","last_became_active":"2024-03-19T01:24:09.903243+0000","last_became_peered":"2024-03-19T01:24:09.903243+0000","last_unstale":"2024-03-19T01:24:11.893572+0000","last_undegraded":"2024-03-19T01:24:11.893572+0000","last_fullsized":"2024-03-19T01:24:11.893572+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:09:03.459117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899557+0000","last_change":"2024-03-19T01:24:10.909772+0000","last_active":"2024-03-19T01:24:11.899557+0000","last_peered":"2024-03-19T01:24:11.899557+0000","last_clean":"2024-03-19T01:24:11.899557+0000","last_became_active":"2024-03-19T01:24:10.900892+0000","last_became_peered":"2024-03-19T01:24:10.900892+0000","last_unstale":"2024-03-19T01:24:11.899557+0000","last_undegraded":"2024-03-19T01:24:11.899557+0000","last_fullsized":"2024-03-19T01:24:11.899557+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:06.571691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256530+0000","last_change":"2024-03-19T01:24:09.896081+0000","last_active":"2024-03-19T01:24:12.256530+0000","last_peered":"2024-03-19T01:24:12.256530+0000","last_clean":"2024-03-19T01:24:12.256530+0000","last_became_active":"2024-03-19T01:24:09.894958+0000","last_became_peered":"2024-03-19T01:24:09.894958+0000","last_unstale":"2024-03-19T01:24:12.256530+0000","last_undegraded":"2024-03-19T01:24:12.256530+0000","last_fullsized":"2024-03-19T01:24:12.256530+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:51.381180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255058+0000","last_change":"2024-03-19T01:24:10.900665+0000","last_active":"2024-03-19T01:24:12.255058+0000","last_peered":"2024-03-19T01:24:12.255058+0000","last_clean":"2024-03-19T01:24:12.255058+0000","last_became_active":"2024-03-19T01:24:10.899936+0000","last_became_peered":"2024-03-19T01:24:10.899936+0000","last_unstale":"2024-03-19T01:24:12.255058+0000","last_undegraded":"2024-03-19T01:24:12.255058+0000","last_fullsized":"2024-03-19T01:24:12.255058+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:31.268081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255400+0000","last_change":"2024-03-19T01:24:09.903637+0000","last_active":"2024-03-19T01:24:12.255400+0000","last_peered":"2024-03-19T01:24:12.255400+0000","last_clean":"2024-03-19T01:24:12.255400+0000","last_became_active":"2024-03-19T01:24:09.902710+0000","last_became_peered":"2024-03-19T01:24:09.902710+0000","last_unstale":"2024-03-19T01:24:12.255400+0000","last_undegraded":"2024-03-19T01:24:12.255400+0000","last_fullsized":"2024-03-19T01:24:12.255400+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:58.982015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257969+0000","last_change":"2024-03-19T01:24:10.933028+0000","last_active":"2024-03-19T01:24:12.257969+0000","last_peered":"2024-03-19T01:24:12.257969+0000","last_clean":"2024-03-19T01:24:12.257969+0000","last_became_active":"2024-03-19T01:24:10.930379+0000","last_became_peered":"2024-03-19T01:24:10.930379+0000","last_unstale":"2024-03-19T01:24:12.257969+0000","last_undegraded":"2024-03-19T01:24:12.257969+0000","last_fullsized":"2024-03-19T01:24:12.257969+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:57:10.997227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898934+0000","last_change":"2024-03-19T01:24:09.905155+0000","last_active":"2024-03-19T01:24:11.898934+0000","last_peered":"2024-03-19T01:24:11.898934+0000","last_clean":"2024-03-19T01:24:11.898934+0000","last_became_active":"2024-03-19T01:24:09.904789+0000","last_became_peered":"2024-03-19T01:24:09.904789+0000","last_unstale":"2024-03-19T01:24:11.898934+0000","last_undegraded":"2024-03-19T01:24:11.898934+0000","last_fullsized":"2024-03-19T01:24:11.898934+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:44.471112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257219+0000","last_change":"2024-03-19T01:24:10.907507+0000","last_active":"2024-03-19T01:24:12.257219+0000","last_peered":"2024-03-19T01:24:12.257219+0000","last_clean":"2024-03-19T01:24:12.257219+0000","last_became_active":"2024-03-19T01:24:10.906586+0000","last_became_peered":"2024-03-19T01:24:10.906586+0000","last_unstale":"2024-03-19T01:24:12.257219+0000","last_undegraded":"2024-03-19T01:24:12.257219+0000","last_fullsized":"2024-03-19T01:24:12.257219+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:18.884118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898584+0000","last_change":"2024-03-19T01:24:09.905939+0000","last_active":"2024-03-19T01:24:11.898584+0000","last_peered":"2024-03-19T01:24:11.898584+0000","last_clean":"2024-03-19T01:24:11.898584+0000","last_became_active":"2024-03-19T01:24:09.905075+0000","last_became_peered":"2024-03-19T01:24:09.905075+0000","last_unstale":"2024-03-19T01:24:11.898584+0000","last_undegraded":"2024-03-19T01:24:11.898584+0000","last_fullsized":"2024-03-19T01:24:11.898584+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:09:03.459117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898782+0000","last_change":"2024-03-19T01:24:10.910309+0000","last_active":"2024-03-19T01:24:11.898782+0000","last_peered":"2024-03-19T01:24:11.898782+0000","last_clean":"2024-03-19T01:24:11.898782+0000","last_became_active":"2024-03-19T01:24:10.904133+0000","last_became_peered":"2024-03-19T01:24:10.904133+0000","last_unstale":"2024-03-19T01:24:11.898782+0000","last_undegraded":"2024-03-19T01:24:11.898782+0000","last_fullsized":"2024-03-19T01:24:11.898782+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:50.681266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256408+0000","last_change":"2024-03-19T01:24:09.904601+0000","last_active":"2024-03-19T01:24:12.256408+0000","last_peered":"2024-03-19T01:24:12.256408+0000","last_clean":"2024-03-19T01:24:12.256408+0000","last_became_active":"2024-03-19T01:24:09.903840+0000","last_became_peered":"2024-03-19T01:24:09.903840+0000","last_unstale":"2024-03-19T01:24:12.256408+0000","last_undegraded":"2024-03-19T01:24:12.256408+0000","last_fullsized":"2024-03-19T01:24:12.256408+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:51.218949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.253752+0000","last_change":"2024-03-19T01:24:10.919208+0000","last_active":"2024-03-19T01:24:12.253752+0000","last_peered":"2024-03-19T01:24:12.253752+0000","last_clean":"2024-03-19T01:24:12.253752+0000","last_became_active":"2024-03-19T01:24:10.918773+0000","last_became_peered":"2024-03-19T01:24:10.918773+0000","last_unstale":"2024-03-19T01:24:12.253752+0000","last_undegraded":"2024-03-19T01:24:12.253752+0000","last_fullsized":"2024-03-19T01:24:12.253752+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:21.718398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898215+0000","last_change":"2024-03-19T01:24:09.909924+0000","last_active":"2024-03-19T01:24:11.898215+0000","last_peered":"2024-03-19T01:24:11.898215+0000","last_clean":"2024-03-19T01:24:11.898215+0000","last_became_active":"2024-03-19T01:24:09.909565+0000","last_became_peered":"2024-03-19T01:24:09.909565+0000","last_unstale":"2024-03-19T01:24:11.898215+0000","last_undegraded":"2024-03-19T01:24:11.898215+0000","last_fullsized":"2024-03-19T01:24:11.898215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:33.389593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.260090+0000","last_change":"2024-03-19T01:24:10.916418+0000","last_active":"2024-03-19T01:24:12.260090+0000","last_peered":"2024-03-19T01:24:12.260090+0000","last_clean":"2024-03-19T01:24:12.260090+0000","last_became_active":"2024-03-19T01:24:10.915682+0000","last_became_peered":"2024-03-19T01:24:10.915682+0000","last_unstale":"2024-03-19T01:24:12.260090+0000","last_undegraded":"2024-03-19T01:24:12.260090+0000","last_fullsized":"2024-03-19T01:24:12.260090+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:04.973168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256780+0000","last_change":"2024-03-19T01:24:09.918125+0000","last_active":"2024-03-19T01:24:12.256780+0000","last_peered":"2024-03-19T01:24:12.256780+0000","last_clean":"2024-03-19T01:24:12.256780+0000","last_became_active":"2024-03-19T01:24:09.917740+0000","last_became_peered":"2024-03-19T01:24:09.917740+0000","last_unstale":"2024-03-19T01:24:12.256780+0000","last_undegraded":"2024-03-19T01:24:12.256780+0000","last_fullsized":"2024-03-19T01:24:12.256780+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:15.428287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256721+0000","last_change":"2024-03-19T01:24:10.900847+0000","last_active":"2024-03-19T01:24:12.256721+0000","last_peered":"2024-03-19T01:24:12.256721+0000","last_clean":"2024-03-19T01:24:12.256721+0000","last_became_active":"2024-03-19T01:24:10.896110+0000","last_became_peered":"2024-03-19T01:24:10.896110+0000","last_unstale":"2024-03-19T01:24:12.256721+0000","last_undegraded":"2024-03-19T01:24:12.256721+0000","last_fullsized":"2024-03-19T01:24:12.256721+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:29.353233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893871+0000","last_change":"2024-03-19T01:24:09.901817+0000","last_active":"2024-03-19T01:24:11.893871+0000","last_peered":"2024-03-19T01:24:11.893871+0000","last_clean":"2024-03-19T01:24:11.893871+0000","last_became_active":"2024-03-19T01:24:09.899759+0000","last_became_peered":"2024-03-19T01:24:09.899759+0000","last_unstale":"2024-03-19T01:24:11.893871+0000","last_undegraded":"2024-03-19T01:24:11.893871+0000","last_fullsized":"2024-03-19T01:24:11.893871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:57.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256372+0000","last_change":"2024-03-19T01:24:10.901428+0000","last_active":"2024-03-19T01:24:12.256372+0000","last_peered":"2024-03-19T01:24:12.256372+0000","last_clean":"2024-03-19T01:24:12.256372+0000","last_became_active":"2024-03-19T01:24:10.901135+0000","last_became_peered":"2024-03-19T01:24:10.901135+0000","last_unstale":"2024-03-19T01:24:12.256372+0000","last_undegraded":"2024-03-19T01:24:12.256372+0000","last_fullsized":"2024-03-19T01:24:12.256372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:31.133305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"29'3","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259604+0000","last_change":"2024-03-19T01:24:09.910513+0000","last_active":"2024-03-19T01:24:12.259604+0000","last_peered":"2024-03-19T01:24:12.259604+0000","last_clean":"2024-03-19T01:24:12.259604+0000","last_became_active":"2024-03-19T01:24:09.909742+0000","last_became_peered":"2024-03-19T01:24:09.909742+0000","last_unstale":"2024-03-19T01:24:12.259604+0000","last_undegraded":"2024-03-19T01:24:12.259604+0000","last_fullsized":"2024-03-19T01:24:12.259604+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:51.381180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259266+0000","last_change":"2024-03-19T01:24:10.920680+0000","last_active":"2024-03-19T01:24:12.259266+0000","last_peered":"2024-03-19T01:24:12.259266+0000","last_clean":"2024-03-19T01:24:12.259266+0000","last_became_active":"2024-03-19T01:24:10.919131+0000","last_became_peered":"2024-03-19T01:24:10.919131+0000","last_unstale":"2024-03-19T01:24:12.259266+0000","last_undegraded":"2024-03-19T01:24:12.259266+0000","last_fullsized":"2024-03-19T01:24:12.259266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:31.133305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"29'6","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256530+0000","last_change":"2024-03-19T01:24:09.897489+0000","last_active":"2024-03-19T01:24:12.256530+0000","last_peered":"2024-03-19T01:24:12.256530+0000","last_clean":"2024-03-19T01:24:12.256530+0000","last_became_active":"2024-03-19T01:24:09.897087+0000","last_became_peered":"2024-03-19T01:24:09.897087+0000","last_unstale":"2024-03-19T01:24:12.256530+0000","last_undegraded":"2024-03-19T01:24:12.256530+0000","last_fullsized":"2024-03-19T01:24:12.256530+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:20.705832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.253762+0000","last_change":"2024-03-19T01:24:10.919116+0000","last_active":"2024-03-19T01:24:12.253762+0000","last_peered":"2024-03-19T01:24:12.253762+0000","last_clean":"2024-03-19T01:24:12.253762+0000","last_became_active":"2024-03-19T01:24:10.918614+0000","last_became_peered":"2024-03-19T01:24:10.918614+0000","last_unstale":"2024-03-19T01:24:12.253762+0000","last_undegraded":"2024-03-19T01:24:12.253762+0000","last_fullsized":"2024-03-19T01:24:12.253762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:50.681266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899001+0000","last_change":"2024-03-19T01:24:09.907779+0000","last_active":"2024-03-19T01:24:11.899001+0000","last_peered":"2024-03-19T01:24:11.899001+0000","last_clean":"2024-03-19T01:24:11.899001+0000","last_became_active":"2024-03-19T01:24:09.906981+0000","last_became_peered":"2024-03-19T01:24:09.906981+0000","last_unstale":"2024-03-19T01:24:11.899001+0000","last_undegraded":"2024-03-19T01:24:11.899001+0000","last_fullsized":"2024-03-19T01:24:11.899001+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:57.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257119+0000","last_change":"2024-03-19T01:24:10.892791+0000","last_active":"2024-03-19T01:24:12.257119+0000","last_peered":"2024-03-19T01:24:12.257119+0000","last_clean":"2024-03-19T01:24:12.257119+0000","last_became_active":"2024-03-19T01:24:10.892506+0000","last_became_peered":"2024-03-19T01:24:10.892506+0000","last_unstale":"2024-03-19T01:24:12.257119+0000","last_undegraded":"2024-03-19T01:24:12.257119+0000","last_fullsized":"2024-03-19T01:24:12.257119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:44:51.202538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.253920+0000","last_change":"2024-03-19T01:24:09.913132+0000","last_active":"2024-03-19T01:24:12.253920+0000","last_peered":"2024-03-19T01:24:12.253920+0000","last_clean":"2024-03-19T01:24:12.253920+0000","last_became_active":"2024-03-19T01:24:09.912821+0000","last_became_peered":"2024-03-19T01:24:09.912821+0000","last_unstale":"2024-03-19T01:24:12.253920+0000","last_undegraded":"2024-03-19T01:24:12.253920+0000","last_fullsized":"2024-03-19T01:24:12.253920+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:23.690868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256656+0000","last_change":"2024-03-19T01:24:10.935675+0000","last_active":"2024-03-19T01:24:12.256656+0000","last_peered":"2024-03-19T01:24:12.256656+0000","last_clean":"2024-03-19T01:24:12.256656+0000","last_became_active":"2024-03-19T01:24:10.934014+0000","last_became_peered":"2024-03-19T01:24:10.934014+0000","last_unstale":"2024-03-19T01:24:12.256656+0000","last_undegraded":"2024-03-19T01:24:12.256656+0000","last_fullsized":"2024-03-19T01:24:12.256656+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:21.075158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256115+0000","last_change":"2024-03-19T01:24:09.916657+0000","last_active":"2024-03-19T01:24:12.256115+0000","last_peered":"2024-03-19T01:24:12.256115+0000","last_clean":"2024-03-19T01:24:12.256115+0000","last_became_active":"2024-03-19T01:24:09.916205+0000","last_became_peered":"2024-03-19T01:24:09.916205+0000","last_unstale":"2024-03-19T01:24:12.256115+0000","last_undegraded":"2024-03-19T01:24:12.256115+0000","last_fullsized":"2024-03-19T01:24:12.256115+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:09.979757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257709+0000","last_change":"2024-03-19T01:24:10.930261+0000","last_active":"2024-03-19T01:24:12.257709+0000","last_peered":"2024-03-19T01:24:12.257709+0000","last_clean":"2024-03-19T01:24:12.257709+0000","last_became_active":"2024-03-19T01:24:10.929567+0000","last_became_peered":"2024-03-19T01:24:10.929567+0000","last_unstale":"2024-03-19T01:24:12.257709+0000","last_undegraded":"2024-03-19T01:24:12.257709+0000","last_fullsized":"2024-03-19T01:24:12.257709+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:51:22.972765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"29'8","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256120+0000","last_change":"2024-03-19T01:24:09.897217+0000","last_active":"2024-03-19T01:24:12.256120+0000","last_peered":"2024-03-19T01:24:12.256120+0000","last_clean":"2024-03-19T01:24:12.256120+0000","last_became_active":"2024-03-19T01:24:09.896713+0000","last_became_peered":"2024-03-19T01:24:09.896713+0000","last_unstale":"2024-03-19T01:24:12.256120+0000","last_undegraded":"2024-03-19T01:24:12.256120+0000","last_fullsized":"2024-03-19T01:24:12.256120+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:15.961961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899956+0000","last_change":"2024-03-19T01:24:10.921859+0000","last_active":"2024-03-19T01:24:11.899956+0000","last_peered":"2024-03-19T01:24:11.899956+0000","last_clean":"2024-03-19T01:24:11.899956+0000","last_became_active":"2024-03-19T01:24:10.921012+0000","last_became_peered":"2024-03-19T01:24:10.921012+0000","last_unstale":"2024-03-19T01:24:11.899956+0000","last_undegraded":"2024-03-19T01:24:11.899956+0000","last_fullsized":"2024-03-19T01:24:11.899956+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.220412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255054+0000","last_change":"2024-03-19T01:24:09.900799+0000","last_active":"2024-03-19T01:24:12.255054+0000","last_peered":"2024-03-19T01:24:12.255054+0000","last_clean":"2024-03-19T01:24:12.255054+0000","last_became_active":"2024-03-19T01:24:09.899109+0000","last_became_peered":"2024-03-19T01:24:09.899109+0000","last_unstale":"2024-03-19T01:24:12.255054+0000","last_undegraded":"2024-03-19T01:24:12.255054+0000","last_fullsized":"2024-03-19T01:24:12.255054+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:37.893827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259311+0000","last_change":"2024-03-19T01:24:10.930718+0000","last_active":"2024-03-19T01:24:12.259311+0000","last_peered":"2024-03-19T01:24:12.259311+0000","last_clean":"2024-03-19T01:24:12.259311+0000","last_became_active":"2024-03-19T01:24:10.930059+0000","last_became_peered":"2024-03-19T01:24:10.930059+0000","last_unstale":"2024-03-19T01:24:12.259311+0000","last_undegraded":"2024-03-19T01:24:12.259311+0000","last_fullsized":"2024-03-19T01:24:12.259311+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:29.353233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899369+0000","last_change":"2024-03-19T01:24:09.907889+0000","last_active":"2024-03-19T01:24:11.899369+0000","last_peered":"2024-03-19T01:24:11.899369+0000","last_clean":"2024-03-19T01:24:11.899369+0000","last_became_active":"2024-03-19T01:24:09.907220+0000","last_became_peered":"2024-03-19T01:24:09.907220+0000","last_unstale":"2024-03-19T01:24:11.899369+0000","last_undegraded":"2024-03-19T01:24:11.899369+0000","last_fullsized":"2024-03-19T01:24:11.899369+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:37.893827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254046+0000","last_change":"2024-03-19T01:24:10.912260+0000","last_active":"2024-03-19T01:24:12.254046+0000","last_peered":"2024-03-19T01:24:12.254046+0000","last_clean":"2024-03-19T01:24:12.254046+0000","last_became_active":"2024-03-19T01:24:10.911887+0000","last_became_peered":"2024-03-19T01:24:10.911887+0000","last_unstale":"2024-03-19T01:24:12.254046+0000","last_undegraded":"2024-03-19T01:24:12.254046+0000","last_fullsized":"2024-03-19T01:24:12.254046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:06.571691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899511+0000","last_change":"2024-03-19T01:24:09.898311+0000","last_active":"2024-03-19T01:24:11.899511+0000","last_peered":"2024-03-19T01:24:11.899511+0000","last_clean":"2024-03-19T01:24:11.899511+0000","last_became_active":"2024-03-19T01:24:09.897568+0000","last_became_peered":"2024-03-19T01:24:09.897568+0000","last_unstale":"2024-03-19T01:24:11.899511+0000","last_undegraded":"2024-03-19T01:24:11.899511+0000","last_fullsized":"2024-03-19T01:24:11.899511+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:45.823952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255141+0000","last_change":"2024-03-19T01:24:10.917973+0000","last_active":"2024-03-19T01:24:12.255141+0000","last_peered":"2024-03-19T01:24:12.255141+0000","last_clean":"2024-03-19T01:24:12.255141+0000","last_became_active":"2024-03-19T01:24:10.917679+0000","last_became_peered":"2024-03-19T01:24:10.917679+0000","last_unstale":"2024-03-19T01:24:12.255141+0000","last_undegraded":"2024-03-19T01:24:12.255141+0000","last_fullsized":"2024-03-19T01:24:12.255141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:28.219770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.899679+0000","last_change":"2024-03-19T01:24:09.908585+0000","last_active":"2024-03-19T01:24:11.899679+0000","last_peered":"2024-03-19T01:24:11.899679+0000","last_clean":"2024-03-19T01:24:11.899679+0000","last_became_active":"2024-03-19T01:24:09.908242+0000","last_became_peered":"2024-03-19T01:24:09.908242+0000","last_unstale":"2024-03-19T01:24:11.899679+0000","last_undegraded":"2024-03-19T01:24:11.899679+0000","last_fullsized":"2024-03-19T01:24:11.899679+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:21.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255082+0000","last_change":"2024-03-19T01:24:10.899196+0000","last_active":"2024-03-19T01:24:12.255082+0000","last_peered":"2024-03-19T01:24:12.255082+0000","last_clean":"2024-03-19T01:24:12.255082+0000","last_became_active":"2024-03-19T01:24:10.898842+0000","last_became_peered":"2024-03-19T01:24:10.898842+0000","last_unstale":"2024-03-19T01:24:12.255082+0000","last_undegraded":"2024-03-19T01:24:12.255082+0000","last_fullsized":"2024-03-19T01:24:12.255082+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:22.112910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.892956+0000","last_change":"2024-03-19T01:24:09.899480+0000","last_active":"2024-03-19T01:24:11.892956+0000","last_peered":"2024-03-19T01:24:11.892956+0000","last_clean":"2024-03-19T01:24:11.892956+0000","last_became_active":"2024-03-19T01:24:09.898875+0000","last_became_peered":"2024-03-19T01:24:09.898875+0000","last_unstale":"2024-03-19T01:24:11.892956+0000","last_undegraded":"2024-03-19T01:24:11.892956+0000","last_fullsized":"2024-03-19T01:24:11.892956+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:37.893827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255799+0000","last_change":"2024-03-19T01:24:10.891031+0000","last_active":"2024-03-19T01:24:12.255799+0000","last_peered":"2024-03-19T01:24:12.255799+0000","last_clean":"2024-03-19T01:24:12.255799+0000","last_became_active":"2024-03-19T01:24:10.890748+0000","last_became_peered":"2024-03-19T01:24:10.890748+0000","last_unstale":"2024-03-19T01:24:12.255799+0000","last_undegraded":"2024-03-19T01:24:12.255799+0000","last_fullsized":"2024-03-19T01:24:12.255799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:58.241465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258008+0000","last_change":"2024-03-19T01:24:09.912771+0000","last_active":"2024-03-19T01:24:12.258008+0000","last_peered":"2024-03-19T01:24:12.258008+0000","last_clean":"2024-03-19T01:24:12.258008+0000","last_became_active":"2024-03-19T01:24:09.912383+0000","last_became_peered":"2024-03-19T01:24:09.912383+0000","last_unstale":"2024-03-19T01:24:12.258008+0000","last_undegraded":"2024-03-19T01:24:12.258008+0000","last_fullsized":"2024-03-19T01:24:12.258008+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:21.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254411+0000","last_change":"2024-03-19T01:24:10.887383+0000","last_active":"2024-03-19T01:24:12.254411+0000","last_peered":"2024-03-19T01:24:12.254411+0000","last_clean":"2024-03-19T01:24:12.254411+0000","last_became_active":"2024-03-19T01:24:10.887141+0000","last_became_peered":"2024-03-19T01:24:10.887141+0000","last_unstale":"2024-03-19T01:24:12.254411+0000","last_undegraded":"2024-03-19T01:24:12.254411+0000","last_fullsized":"2024-03-19T01:24:12.254411+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:34.402159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898215+0000","last_change":"2024-03-19T01:24:09.890802+0000","last_active":"2024-03-19T01:24:11.898215+0000","last_peered":"2024-03-19T01:24:11.898215+0000","last_clean":"2024-03-19T01:24:11.898215+0000","last_became_active":"2024-03-19T01:24:09.889918+0000","last_became_peered":"2024-03-19T01:24:09.889918+0000","last_unstale":"2024-03-19T01:24:11.898215+0000","last_undegraded":"2024-03-19T01:24:11.898215+0000","last_fullsized":"2024-03-19T01:24:11.898215+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:58.773246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256194+0000","last_change":"2024-03-19T01:24:10.936768+0000","last_active":"2024-03-19T01:24:12.256194+0000","last_peered":"2024-03-19T01:24:12.256194+0000","last_clean":"2024-03-19T01:24:12.256194+0000","last_became_active":"2024-03-19T01:24:10.936087+0000","last_became_peered":"2024-03-19T01:24:10.936087+0000","last_unstale":"2024-03-19T01:24:12.256194+0000","last_undegraded":"2024-03-19T01:24:12.256194+0000","last_fullsized":"2024-03-19T01:24:12.256194+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:41.581399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256753+0000","last_change":"2024-03-19T01:24:09.918453+0000","last_active":"2024-03-19T01:24:12.256753+0000","last_peered":"2024-03-19T01:24:12.256753+0000","last_clean":"2024-03-19T01:24:12.256753+0000","last_became_active":"2024-03-19T01:24:09.917806+0000","last_became_peered":"2024-03-19T01:24:09.917806+0000","last_unstale":"2024-03-19T01:24:12.256753+0000","last_undegraded":"2024-03-19T01:24:12.256753+0000","last_fullsized":"2024-03-19T01:24:12.256753+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:09.979757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257564+0000","last_change":"2024-03-19T01:24:10.920280+0000","last_active":"2024-03-19T01:24:12.257564+0000","last_peered":"2024-03-19T01:24:12.257564+0000","last_clean":"2024-03-19T01:24:12.257564+0000","last_became_active":"2024-03-19T01:24:10.919985+0000","last_became_peered":"2024-03-19T01:24:10.919985+0000","last_unstale":"2024-03-19T01:24:12.257564+0000","last_undegraded":"2024-03-19T01:24:12.257564+0000","last_fullsized":"2024-03-19T01:24:12.257564+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:38.906393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.255184+0000","last_change":"2024-03-19T01:24:09.916657+0000","last_active":"2024-03-19T01:24:12.255184+0000","last_peered":"2024-03-19T01:24:12.255184+0000","last_clean":"2024-03-19T01:24:12.255184+0000","last_became_active":"2024-03-19T01:24:09.916140+0000","last_became_peered":"2024-03-19T01:24:09.916140+0000","last_unstale":"2024-03-19T01:24:12.255184+0000","last_undegraded":"2024-03-19T01:24:12.255184+0000","last_fullsized":"2024-03-19T01:24:12.255184+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:41.797307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.259496+0000","last_change":"2024-03-19T01:24:10.915155+0000","last_active":"2024-03-19T01:24:12.259496+0000","last_peered":"2024-03-19T01:24:12.259496+0000","last_clean":"2024-03-19T01:24:12.259496+0000","last_became_active":"2024-03-19T01:24:10.914437+0000","last_became_peered":"2024-03-19T01:24:10.914437+0000","last_unstale":"2024-03-19T01:24:12.259496+0000","last_undegraded":"2024-03-19T01:24:12.259496+0000","last_fullsized":"2024-03-19T01:24:12.259496+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:02.824816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893841+0000","last_change":"2024-03-19T01:24:09.904351+0000","last_active":"2024-03-19T01:24:11.893841+0000","last_peered":"2024-03-19T01:24:11.893841+0000","last_clean":"2024-03-19T01:24:11.893841+0000","last_became_active":"2024-03-19T01:24:09.902472+0000","last_became_peered":"2024-03-19T01:24:09.902472+0000","last_unstale":"2024-03-19T01:24:11.893841+0000","last_undegraded":"2024-03-19T01:24:11.893841+0000","last_fullsized":"2024-03-19T01:24:11.893841+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:24:44.471112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893060+0000","last_change":"2024-03-19T01:24:10.893855+0000","last_active":"2024-03-19T01:24:11.893060+0000","last_peered":"2024-03-19T01:24:11.893060+0000","last_clean":"2024-03-19T01:24:11.893060+0000","last_became_active":"2024-03-19T01:24:10.893531+0000","last_became_peered":"2024-03-19T01:24:10.893531+0000","last_unstale":"2024-03-19T01:24:11.893060+0000","last_undegraded":"2024-03-19T01:24:11.893060+0000","last_fullsized":"2024-03-19T01:24:11.893060+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:16.974528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.254558+0000","last_change":"2024-03-19T01:24:09.901870+0000","last_active":"2024-03-19T01:24:12.254558+0000","last_peered":"2024-03-19T01:24:12.254558+0000","last_clean":"2024-03-19T01:24:12.254558+0000","last_became_active":"2024-03-19T01:24:09.899450+0000","last_became_peered":"2024-03-19T01:24:09.899450+0000","last_unstale":"2024-03-19T01:24:12.254558+0000","last_undegraded":"2024-03-19T01:24:12.254558+0000","last_fullsized":"2024-03-19T01:24:12.254558+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:05.746340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258160+0000","last_change":"2024-03-19T01:24:10.915432+0000","last_active":"2024-03-19T01:24:12.258160+0000","last_peered":"2024-03-19T01:24:12.258160+0000","last_clean":"2024-03-19T01:24:12.258160+0000","last_became_active":"2024-03-19T01:24:10.914630+0000","last_became_peered":"2024-03-19T01:24:10.914630+0000","last_unstale":"2024-03-19T01:24:12.258160+0000","last_undegraded":"2024-03-19T01:24:12.258160+0000","last_fullsized":"2024-03-19T01:24:12.258160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:52:34.148842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.893189+0000","last_change":"2024-03-19T01:24:09.916012+0000","last_active":"2024-03-19T01:24:11.893189+0000","last_peered":"2024-03-19T01:24:11.893189+0000","last_clean":"2024-03-19T01:24:11.893189+0000","last_became_active":"2024-03-19T01:24:09.915593+0000","last_became_peered":"2024-03-19T01:24:09.915593+0000","last_unstale":"2024-03-19T01:24:11.893189+0000","last_undegraded":"2024-03-19T01:24:11.893189+0000","last_fullsized":"2024-03-19T01:24:11.893189+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:41.797307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.892983+0000","last_change":"2024-03-19T01:24:10.910324+0000","last_active":"2024-03-19T01:24:11.892983+0000","last_peered":"2024-03-19T01:24:11.892983+0000","last_clean":"2024-03-19T01:24:11.892983+0000","last_became_active":"2024-03-19T01:24:10.910029+0000","last_became_peered":"2024-03-19T01:24:10.910029+0000","last_unstale":"2024-03-19T01:24:11.892983+0000","last_undegraded":"2024-03-19T01:24:11.892983+0000","last_fullsized":"2024-03-19T01:24:11.892983+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:23.319987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.253824+0000","last_change":"2024-03-19T01:24:09.916901+0000","last_active":"2024-03-19T01:24:12.253824+0000","last_peered":"2024-03-19T01:24:12.253824+0000","last_clean":"2024-03-19T01:24:12.253824+0000","last_became_active":"2024-03-19T01:24:09.916155+0000","last_became_peered":"2024-03-19T01:24:09.916155+0000","last_unstale":"2024-03-19T01:24:12.253824+0000","last_undegraded":"2024-03-19T01:24:12.253824+0000","last_fullsized":"2024-03-19T01:24:12.253824+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:51:21.960199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256091+0000","last_change":"2024-03-19T01:24:10.894775+0000","last_active":"2024-03-19T01:24:12.256091+0000","last_peered":"2024-03-19T01:24:12.256091+0000","last_clean":"2024-03-19T01:24:12.256091+0000","last_became_active":"2024-03-19T01:24:10.894372+0000","last_became_peered":"2024-03-19T01:24:10.894372+0000","last_unstale":"2024-03-19T01:24:12.256091+0000","last_undegraded":"2024-03-19T01:24:12.256091+0000","last_fullsized":"2024-03-19T01:24:12.256091+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:15:18.884118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898621+0000","last_change":"2024-03-19T01:24:09.916961+0000","last_active":"2024-03-19T01:24:11.898621+0000","last_peered":"2024-03-19T01:24:11.898621+0000","last_clean":"2024-03-19T01:24:11.898621+0000","last_became_active":"2024-03-19T01:24:09.916341+0000","last_became_peered":"2024-03-19T01:24:09.916341+0000","last_unstale":"2024-03-19T01:24:11.898621+0000","last_undegraded":"2024-03-19T01:24:11.898621+0000","last_fullsized":"2024-03-19T01:24:11.898621+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:27.207204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:11.898472+0000","last_change":"2024-03-19T01:24:10.902504+0000","last_active":"2024-03-19T01:24:11.898472+0000","last_peered":"2024-03-19T01:24:11.898472+0000","last_clean":"2024-03-19T01:24:11.898472+0000","last_became_active":"2024-03-19T01:24:10.900667+0000","last_became_peered":"2024-03-19T01:24:10.900667+0000","last_unstale":"2024-03-19T01:24:11.898472+0000","last_undegraded":"2024-03-19T01:24:11.898472+0000","last_fullsized":"2024-03-19T01:24:11.898472+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:52.393746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256016+0000","last_change":"2024-03-19T01:24:09.912281+0000","last_active":"2024-03-19T01:24:12.256016+0000","last_peered":"2024-03-19T01:24:12.256016+0000","last_clean":"2024-03-19T01:24:12.256016+0000","last_became_active":"2024-03-19T01:24:09.911745+0000","last_became_peered":"2024-03-19T01:24:09.911745+0000","last_unstale":"2024-03-19T01:24:12.256016+0000","last_undegraded":"2024-03-19T01:24:12.256016+0000","last_fullsized":"2024-03-19T01:24:12.256016+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:22.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.258025+0000","last_change":"2024-03-19T01:24:10.925298+0000","last_active":"2024-03-19T01:24:12.258025+0000","last_peered":"2024-03-19T01:24:12.258025+0000","last_clean":"2024-03-19T01:24:12.258025+0000","last_became_active":"2024-03-19T01:24:10.924512+0000","last_became_peered":"2024-03-19T01:24:10.924512+0000","last_unstale":"2024-03-19T01:24:12.258025+0000","last_undegraded":"2024-03-19T01:24:12.258025+0000","last_fullsized":"2024-03-19T01:24:12.258025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:09.863743+0000","last_clean_scrub_stamp":"2024-03-19T01:24:09.863743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:25.852042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.256403+0000","last_change":"2024-03-19T01:24:09.912688+0000","last_active":"2024-03-19T01:24:12.256403+0000","last_peered":"2024-03-19T01:24:12.256403+0000","last_clean":"2024-03-19T01:24:12.256403+0000","last_became_active":"2024-03-19T01:24:09.911954+0000","last_became_peered":"2024-03-19T01:24:09.911954+0000","last_unstale":"2024-03-19T01:24:12.256403+0000","last_undegraded":"2024-03-19T01:24:12.256403+0000","last_fullsized":"2024-03-19T01:24:12.256403+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:24:08.851177+0000","last_clean_scrub_stamp":"2024-03-19T01:24:08.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:33.389593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":187,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T01:24:12.257190+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:24:12.257190+0000","last_peered":"2024-03-19T01:24:12.257190+0000","last_clean":"2024-03-19T01:24:12.257190+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:24:12.257190+0000","last_undegraded":"2024-03-19T01:24:12.257190+0000","last_fullsized":"2024-03-19T01:24:12.257190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":933888,"data_stored":719141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":344064,"data_stored":129299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":344064,"data_stored":129355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":344064,"data_stored":128787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":130317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":339968,"data_stored":128753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":130317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":929792,"data_stored":719051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:24:15.323 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-19T01:24:15.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-19T01:24:15.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9943734640 1 Processor -- start 2024-03-19T01:24:15.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9943734640 1 -- start start 2024-03-19T01:24:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9943734640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f993c0b4a90 0x7f993c150b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9943734640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f993c0b2c20 con 0x7f993c0b56e0 2024-03-19T01:24:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9943734640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f993c0b5d00 con 0x7f993c0b77f0 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9943734640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f993c0b5fe0 con 0x7f993c0b4a90 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9940ccf640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f993c0b56e0 0x7f993c0b4380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37674/0 (socket says 172.21.15.7:37674) 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9940ccf640 1 -- 172.21.15.7:0/2311976064 learned_addr learned my addr 172.21.15.7:0/2311976064 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9941cd1640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f993c0b4a90 0x7f993c150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1818969844 0 0) 0x7f993c0b2c20 con 0x7f993c0b56e0 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9924003530 con 0x7f993c0b56e0 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2725430748 0 0) 0x7f9924003530 con 0x7f993c0b56e0 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f993c0b4a90 msgr2=0x7f993c150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f993c0b4a90 0x7f993c150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 >> v1:172.21.15.167:6789/0 conn(0x7f993c0b77f0 legacy=0x7f993c0b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993c0b64d0 con 0x7f993c0b56e0 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f9930001680 con 0x7f993c0b56e0 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 >> v1:172.21.15.7:6789/0 conn(0x7f993c0b56e0 legacy=0x7f993c0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99240039f0 0x7f9924003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f993c0b4a90 0x7f9924004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9924004d50 0x7f99240051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f993c0b5fe0 con 0x7f99240039f0 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f992400bb00 con 0x7f9924004d50 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f992400bd00 con 0x7f993c0b4a90 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f99414d0640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99240039f0 0x7f9924003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9940ccf640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f993c0b4a90 0x7f9924004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f9941cd1640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9924004d50 0x7f99240051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f99414d0640 1 -- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f993c0b4a90 msgr2=0x7f9924004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f99414d0640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f993c0b4a90 0x7f9924004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.498+0000 7f99414d0640 1 -- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9924004d50 msgr2=0x7f99240051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f99414d0640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9924004d50 0x7f99240051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f99414d0640 1 -- 172.21.15.7:0/2311976064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9924003530 con 0x7f99240039f0 2024-03-19T01:24:15.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f99414d0640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99240039f0 0x7f9924003e10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9934001780 tx=0x7f993401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b3d3940d4215a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:15.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99340013a0 con 0x7f99240039f0 2024-03-19T01:24:15.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99340052f0 con 0x7f99240039f0 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f993bfff640 1 -- 172.21.15.7:0/2311976064 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9934003840 con 0x7f99240039f0 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f9943734640 1 -- 172.21.15.7:0/2311976064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99240039f0 msgr2=0x7f9924003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f9943734640 1 --2- 172.21.15.7:0/2311976064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99240039f0 0x7f9924003e10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9934001780 tx=0x7f993401c360 comp rx=0 tx=0).stop 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f9941cd1640 1 -- 172.21.15.7:0/2311976064 reap_dead start 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f9943734640 1 -- 172.21.15.7:0/2311976064 shutdown_connections 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f9943734640 1 -- 172.21.15.7:0/2311976064 >> 172.21.15.7:0/2311976064 conn(0x7f993c06b2c0 msgr2=0x7f993c06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f9943734640 1 -- 172.21.15.7:0/2311976064 shutdown_connections 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f9943734640 1 -- 172.21.15.7:0/2311976064 wait complete. 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.502+0000 7f9943734640 1 Processor -- start 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 -- start start 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99240039f0 0x7f993c13afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993c1e3e70 0x7f993c0b9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f993c1dfcf0 0x7f993c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f993c151500 con 0x7f993c1e3e70 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f993c151380 con 0x7f993c1dfcf0 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f993c151680 con 0x7f99240039f0 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9940ccf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993c1e3e70 0x7f993c0b9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9940ccf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993c1e3e70 0x7f993c0b9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33900/0 (socket says 172.21.15.7:33900) 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9940ccf640 1 -- 172.21.15.7:0/2507080323 learned_addr learned my addr 172.21.15.7:0/2507080323 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f99414d0640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99240039f0 0x7f993c13afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9941cd1640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f993c1dfcf0 0x7f993c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9940ccf640 1 -- 172.21.15.7:0/2507080323 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99240039f0 msgr2=0x7f993c13afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9940ccf640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99240039f0 0x7f993c13afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9940ccf640 1 -- 172.21.15.7:0/2507080323 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f993c1dfcf0 msgr2=0x7f993c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9940ccf640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f993c1dfcf0 0x7f993c06d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9940ccf640 1 -- 172.21.15.7:0/2507080323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993c1e3390 con 0x7f993c1e3e70 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9940ccf640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993c1e3e70 0x7f993c0b9de0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f993000e800 tx=0x7f993000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9939ffb640 1 -- 172.21.15.7:0/2507080323 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993000ee50 con 0x7f993c1e3e70 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9939ffb640 1 -- 172.21.15.7:0/2507080323 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9930018070 con 0x7f993c1e3e70 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f993c1e3670 con 0x7f993c1e3e70 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9939ffb640 1 -- 172.21.15.7:0/2507080323 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993000cab0 con 0x7f993c1e3e70 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f993c13b840 con 0x7f993c1e3e70 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f993c075570 con 0x7f993c1e3e70 2024-03-19T01:24:15.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9939ffb640 1 -- 172.21.15.7:0/2507080323 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f993000cc50 con 0x7f993c1e3e70 2024-03-19T01:24:15.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9939ffb640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9928067830 0x7f9928069cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f9939ffb640 1 -- 172.21.15.7:0/2507080323 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f9930086130 con 0x7f993c1e3e70 2024-03-19T01:24:15.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.506+0000 7f99414d0640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9928067830 0x7f9928069cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.510+0000 7f99414d0640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9928067830 0x7f9928069cf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9934001750 tx=0x7f99340011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:15.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.510+0000 7f9939ffb640 1 -- 172.21.15.7:0/2507080323 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f993001a310 con 0x7f993c1e3e70 2024-03-19T01:24:15.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.722+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f993c071c80 con 0x7f993c1e3e70 2024-03-19T01:24:15.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9939ffb640 1 -- 172.21.15.7:0/2507080323 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f9930056f20 con 0x7f993c1e3e70 2024-03-19T01:24:15.728 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-03-19T01:24:15.728 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDR6Phl6xj5FxAAeVMK9nlnMPrcDkxprwQY2w== 2024-03-19T01:24:15.728 INFO:teuthology.orchestra.run.smithi007.stdout: caps mds = "allow" 2024-03-19T01:24:15.728 INFO:teuthology.orchestra.run.smithi007.stdout: caps mon = "allow r" 2024-03-19T01:24:15.728 INFO:teuthology.orchestra.run.smithi007.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-19T01:24:15.728 INFO:teuthology.orchestra.run.smithi007.stderr:updated caps for client.0 2024-03-19T01:24:15.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9928067830 msgr2=0x7f9928069cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9928067830 0x7f9928069cf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9934001750 tx=0x7f99340011a0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993c1e3e70 msgr2=0x7f993c0b9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993c1e3e70 0x7f993c0b9de0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f993000e800 tx=0x7f993000ec00 comp rx=0 tx=0).stop 2024-03-19T01:24:15.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 shutdown_connections 2024-03-19T01:24:15.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9928067830 0x7f9928069cf0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f993c1dfcf0 0x7f993c06d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993c1e3e70 0x7f993c0b9de0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 --2- 172.21.15.7:0/2507080323 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99240039f0 0x7f993c13afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 >> 172.21.15.7:0/2507080323 conn(0x7f993c06b2c0 msgr2=0x7f993c138070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:15.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 shutdown_connections 2024-03-19T01:24:15.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.726+0000 7f9943734640 1 -- 172.21.15.7:0/2507080323 wait complete. 2024-03-19T01:24:15.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:24:15.852 INFO:tasks.ceph.mds.a.smithi007.stderr:2024-03-19T01:24:15.850+0000 7f7c142e6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T01:24:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2fe80640 1 Processor -- start 2024-03-19T01:24:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2fe80640 1 -- start start 2024-03-19T01:24:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2fe80640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f9e28067260 0x7f9e28145a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2fe80640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28145f50 0x7f9e28148950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2fe80640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e28067640 con 0x7f9e28148e90 2024-03-19T01:24:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2fe80640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f9e28145580 con 0x7f9e28067260 2024-03-19T01:24:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2fe80640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9e28146390 con 0x7f9e28145f50 2024-03-19T01:24:15.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2d41b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28145f50 0x7f9e28148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2dc1c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f9e28067260 0x7f9e28145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2dc1c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f9e28067260 0x7f9e28145a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:55742/0 (socket says 172.21.15.7:55742) 2024-03-19T01:24:15.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2d41b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28145f50 0x7f9e28148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:39506/0 (socket says 172.21.15.7:39506) 2024-03-19T01:24:15.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2dc1c640 1 -- 172.21.15.7:0/831137030 learned_addr learned my addr 172.21.15.7:0/831137030 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:15.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2e41d640 1 --1- 172.21.15.7:0/831137030 >> v1:172.21.15.167:6789/0 conn(0x7f9e28148e90 0x7f9e2813ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:58600/0 (socket says 172.21.15.7:58600) 2024-03-19T01:24:15.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2d41b640 1 -- 172.21.15.7:0/831137030 >> v2:172.21.15.167:3300/0 conn(0x7f9e28067260 msgr2=0x7f9e28145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:15.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2d41b640 1 --2- 172.21.15.7:0/831137030 >> v2:172.21.15.167:3300/0 conn(0x7f9e28067260 0x7f9e28145a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2d41b640 1 -- 172.21.15.7:0/831137030 >> v1:172.21.15.167:6789/0 conn(0x7f9e28148e90 legacy=0x7f9e2813ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2d41b640 1 -- 172.21.15.7:0/831137030 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e28070090 con 0x7f9e28145f50 2024-03-19T01:24:15.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.858+0000 7f9e2dc1c640 1 --2- 172.21.15.7:0/831137030 >> v2:172.21.15.167:3300/0 conn(0x7f9e28067260 0x7f9e28145a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:15.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2d41b640 1 --2- 172.21.15.7:0/831137030 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28145f50 0x7f9e28148950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9e28145700 tx=0x7f9e1801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb35f67ababd812 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2cc1a640 1 -- 172.21.15.7:0/831137030 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e180b0070 con 0x7f9e28145f50 2024-03-19T01:24:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2cc1a640 1 -- 172.21.15.7:0/831137030 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e180ab600 con 0x7f9e28145f50 2024-03-19T01:24:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2cc1a640 1 -- 172.21.15.7:0/831137030 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e180ab9d0 con 0x7f9e28145f50 2024-03-19T01:24:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 -- 172.21.15.7:0/831137030 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28145f50 msgr2=0x7f9e28148950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 --2- 172.21.15.7:0/831137030 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28145f50 0x7f9e28148950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9e28145700 tx=0x7f9e1801c410 comp rx=0 tx=0).stop 2024-03-19T01:24:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 -- 172.21.15.7:0/831137030 shutdown_connections 2024-03-19T01:24:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 --2- 172.21.15.7:0/831137030 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28145f50 0x7f9e28148950 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 --2- 172.21.15.7:0/831137030 >> v2:172.21.15.167:3300/0 conn(0x7f9e28067260 0x7f9e28145a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 -- 172.21.15.7:0/831137030 >> 172.21.15.7:0/831137030 conn(0x7f9e280b9190 msgr2=0x7f9e280b95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 -- 172.21.15.7:0/831137030 shutdown_connections 2024-03-19T01:24:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 -- 172.21.15.7:0/831137030 wait complete. 2024-03-19T01:24:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 Processor -- start 2024-03-19T01:24:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 -- start start 2024-03-19T01:24:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e28067260 0x7f9e280b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e28145f50 0x7f9e280b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28148e90 0x7f9e28146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9e280b5490 con 0x7f9e28067260 2024-03-19T01:24:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9e280b5310 con 0x7f9e28145f50 2024-03-19T01:24:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.862+0000 7f9e2fe80640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9e280b5920 con 0x7f9e28148e90 2024-03-19T01:24:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2dc1c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e28067260 0x7f9e280b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2dc1c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e28067260 0x7f9e280b1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33904/0 (socket says 172.21.15.7:33904) 2024-03-19T01:24:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2dc1c640 1 -- 172.21.15.7:0/3882094388 learned_addr learned my addr 172.21.15.7:0/3882094388 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2e41d640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28148e90 0x7f9e28146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2d41b640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e28145f50 0x7f9e280b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2dc1c640 1 -- 172.21.15.7:0/3882094388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28148e90 msgr2=0x7f9e28146540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2dc1c640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28148e90 0x7f9e28146540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2dc1c640 1 -- 172.21.15.7:0/3882094388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e28145f50 msgr2=0x7f9e280b7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2dc1c640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e28145f50 0x7f9e280b7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2dc1c640 1 -- 172.21.15.7:0/3882094388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2813e670 con 0x7f9e28067260 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2e41d640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28148e90 0x7f9e28146540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2dc1c640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e28067260 0x7f9e280b1270 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400efc0 tx=0x7f9e2400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e1e7fc640 1 -- 172.21.15.7:0/3882094388 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2400ce60 con 0x7f9e28067260 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e1e7fc640 1 -- 172.21.15.7:0/3882094388 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e24019070 con 0x7f9e28067260 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e280b17b0 con 0x7f9e28067260 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e1e7fc640 1 -- 172.21.15.7:0/3882094388 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24014680 con 0x7f9e28067260 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e280b8e50 con 0x7f9e28067260 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e28075460 con 0x7f9e28067260 2024-03-19T01:24:15.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e1e7fc640 1 -- 172.21.15.7:0/3882094388 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9e24012050 con 0x7f9e28067260 2024-03-19T01:24:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e1e7fc640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9e0c067660 0x7f9e0c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2d41b640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9e0c067660 0x7f9e0c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e1e7fc640 1 -- 172.21.15.7:0/3882094388 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f9e240868d0 con 0x7f9e28067260 2024-03-19T01:24:15.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.866+0000 7f9e2d41b640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9e0c067660 0x7f9e0c069b20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9e180ad000 tx=0x7f9e18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:15.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:15.870+0000 7f9e1e7fc640 1 -- 172.21.15.7:0/3882094388 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9e24010040 con 0x7f9e28067260 2024-03-19T01:24:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.058+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e280b0e90 con 0x7f9e28067260 2024-03-19T01:24:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.058+0000 7f9e1e7fc640 1 -- 172.21.15.7:0/3882094388 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3531 (secure 0 0 0) 0x7f9e24057540 con 0x7f9e28067260 2024-03-19T01:24:16.063 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:16.063 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4753,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/3744754940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3744754940},{"type":"v1","addr":"172.21.15.167:6833","nonce":3744754940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:24:10.251824+0000","modified":"2024-03-19T01:24:13.280725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5001":{"gid":5001,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/4260183338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4260183338},{"type":"v1","addr":"172.21.15.7:6835","nonce":4260183338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5001,"qdb_cluster":[5001]},"id":2}]} 2024-03-19T01:24:16.063 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 16 2024-03-19T01:24:16.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9e0c067660 msgr2=0x7f9e0c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9e0c067660 0x7f9e0c069b20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9e180ad000 tx=0x7f9e18008040 comp rx=0 tx=0).stop 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e28067260 msgr2=0x7f9e280b1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e28067260 0x7f9e280b1270 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400efc0 tx=0x7f9e2400c490 comp rx=0 tx=0).stop 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 shutdown_connections 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9e0c067660 0x7f9e0c069b20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e28148e90 0x7f9e28146540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e28145f50 0x7f9e280b7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 --2- 172.21.15.7:0/3882094388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e28067260 0x7f9e280b1270 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 >> 172.21.15.7:0/3882094388 conn(0x7f9e280b9190 msgr2=0x7f9e2813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 shutdown_connections 2024-03-19T01:24:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.062+0000 7f9e2fe80640 1 -- 172.21.15.7:0/3882094388 wait complete. 2024-03-19T01:24:16.281 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:24:10.251824+0000', 'modified': '2024-03-19T01:24:13.280725+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5001}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5001': {'gid': 5001, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.7:6835/4260183338', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6834', 'nonce': 4260183338}, {'type': 'v1', 'addr': '172.21.15.7:6835', 'nonce': 4260183338}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5001, 'qdb_cluster': [5001]} 2024-03-19T01:24:16.281 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-19T01:24:16.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-19T01:24:16.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b5aec7640 1 Processor -- start 2024-03-19T01:24:16.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b5aec7640 1 -- start start 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b5aec7640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0b540b9ed0 0x7f0b541411d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b5aec7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0b540b47b0 0x7f0b5414a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b5aec7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b540ba2b0 con 0x7f0b54141710 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b5aec7640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f0b540b6cc0 con 0x7f0b540b47b0 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b5aec7640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f0b540b52b0 con 0x7f0b540b9ed0 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b53fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0b540b47b0 0x7f0b5414a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b59464640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0b54141710 0x7f0b540b7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37676/0 (socket says 172.21.15.7:37676) 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b59464640 1 -- 172.21.15.7:0/591584838 learned_addr learned my addr 172.21.15.7:0/591584838 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b58c63640 1 --2- 172.21.15.7:0/591584838 >> v2:172.21.15.167:3300/0 conn(0x7f0b540b9ed0 0x7f0b541411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b537fe640 1 -- 172.21.15.7:0/591584838 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278906867 0 0) 0x7f0b540ba2b0 con 0x7f0b54141710 2024-03-19T01:24:16.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b537fe640 1 -- 172.21.15.7:0/591584838 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b3c003530 con 0x7f0b54141710 2024-03-19T01:24:16.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b53fff640 1 -- 172.21.15.7:0/591584838 >> v2:172.21.15.167:3300/0 conn(0x7f0b540b9ed0 msgr2=0x7f0b541411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b53fff640 1 --2- 172.21.15.7:0/591584838 >> v2:172.21.15.167:3300/0 conn(0x7f0b540b9ed0 0x7f0b541411d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b53fff640 1 -- 172.21.15.7:0/591584838 >> v1:172.21.15.7:6789/0 conn(0x7f0b54141710 legacy=0x7f0b540b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.410+0000 7f0b53fff640 1 -- 172.21.15.7:0/591584838 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b540b5590 con 0x7f0b540b47b0 2024-03-19T01:24:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b53fff640 1 --2- 172.21.15.7:0/591584838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b540b47b0 0x7f0b5414a9b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0b440097c0 tx=0x7f0b4401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc99b29a2ad68567 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b537fe640 1 -- 172.21.15.7:0/591584838 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b440b0070 con 0x7f0b540b47b0 2024-03-19T01:24:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b537fe640 1 -- 172.21.15.7:0/591584838 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b440ab600 con 0x7f0b540b47b0 2024-03-19T01:24:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b537fe640 1 -- 172.21.15.7:0/591584838 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b440ab9d0 con 0x7f0b540b47b0 2024-03-19T01:24:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 -- 172.21.15.7:0/591584838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b540b47b0 msgr2=0x7f0b5414a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 --2- 172.21.15.7:0/591584838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b540b47b0 0x7f0b5414a9b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0b440097c0 tx=0x7f0b4401c310 comp rx=0 tx=0).stop 2024-03-19T01:24:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 -- 172.21.15.7:0/591584838 shutdown_connections 2024-03-19T01:24:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 --2- 172.21.15.7:0/591584838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b540b47b0 0x7f0b5414a9b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 --2- 172.21.15.7:0/591584838 >> v2:172.21.15.167:3300/0 conn(0x7f0b540b9ed0 0x7f0b541411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 -- 172.21.15.7:0/591584838 >> 172.21.15.7:0/591584838 conn(0x7f0b54072df0 msgr2=0x7f0b54073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 -- 172.21.15.7:0/591584838 shutdown_connections 2024-03-19T01:24:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 -- 172.21.15.7:0/591584838 wait complete. 2024-03-19T01:24:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 Processor -- start 2024-03-19T01:24:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.414+0000 7f0b5aec7640 1 -- start start 2024-03-19T01:24:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b5aec7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b540b47b0 0x7f0b541451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b5aec7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b540b9ed0 0x7f0b54142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b5aec7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54141710 0x7f0b54147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b5aec7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0b540b4d80 con 0x7f0b54141710 2024-03-19T01:24:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b5aec7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0b540b4c00 con 0x7f0b540b47b0 2024-03-19T01:24:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b5aec7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0b54154cd0 con 0x7f0b540b9ed0 2024-03-19T01:24:16.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b59464640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54141710 0x7f0b54147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b59464640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54141710 0x7f0b54147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33926/0 (socket says 172.21.15.7:33926) 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b59464640 1 -- 172.21.15.7:0/1168686435 learned_addr learned my addr 172.21.15.7:0/1168686435 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b58c63640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b540b47b0 0x7f0b541451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b53fff640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b540b9ed0 0x7f0b54142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b59464640 1 -- 172.21.15.7:0/1168686435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b540b9ed0 msgr2=0x7f0b54142da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b59464640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b540b9ed0 0x7f0b54142da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b59464640 1 -- 172.21.15.7:0/1168686435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b540b47b0 msgr2=0x7f0b541451a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b59464640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b540b47b0 0x7f0b541451a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b59464640 1 -- 172.21.15.7:0/1168686435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b54143f40 con 0x7f0b54141710 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b59464640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54141710 0x7f0b54147d90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00caa0 tx=0x7f0b4c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b517fa640 1 -- 172.21.15.7:0/1168686435 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c017070 con 0x7f0b54141710 2024-03-19T01:24:16.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b517fa640 1 -- 172.21.15.7:0/1168686435 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b4c00aba0 con 0x7f0b54141710 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b5aec7640 1 -- 172.21.15.7:0/1168686435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b54146f90 con 0x7f0b54141710 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b517fa640 1 -- 172.21.15.7:0/1168686435 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c012400 con 0x7f0b54141710 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b5aec7640 1 -- 172.21.15.7:0/1168686435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b541425a0 con 0x7f0b54141710 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.418+0000 7f0b517fa640 1 -- 172.21.15.7:0/1168686435 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0b4c01a070 con 0x7f0b54141710 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b517fa640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0b40067690 0x7f0b40069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0b40067690 0x7f0b40069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b517fa640 1 -- 172.21.15.7:0/1168686435 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f0b4c085600 con 0x7f0b54141710 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0b40067690 0x7f0b40069b50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009af0 tx=0x7f0b48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 Processor -- start 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 -- start start 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6de10 0x55e72eb6e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6e730 0x55e72eb73280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72eb73ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e72ea2a280 con 0x55e72eb6de10 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55e72e992690 con 0x55e72eb6e730 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55e72e919c50 con 0x55e72eb737c0 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6de10 0x55e72eb6e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6de10 0x55e72eb6e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33934/0 (socket says 172.21.15.7:33934) 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 -- 172.21.15.7:0/2684084988 learned_addr learned my addr 172.21.15.7:0/2684084988 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b59464640 1 --2- 172.21.15.7:0/2684084988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72eb73ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b53fff640 1 --2- 172.21.15.7:0/2684084988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6e730 0x55e72eb73280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 -- 172.21.15.7:0/2684084988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 msgr2=0x55e72eb73ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 --2- 172.21.15.7:0/2684084988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72eb73ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 -- 172.21.15.7:0/2684084988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6e730 msgr2=0x55e72eb73280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 --2- 172.21.15.7:0/2684084988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6e730 0x55e72eb73280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 -- 172.21.15.7:0/2684084988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e72e919ea0 con 0x55e72eb6de10 2024-03-19T01:24:16.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b58c63640 1 --2- 172.21.15.7:0/2684084988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6de10 0x55e72eb6e1f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0b48011dc0 tx=0x7f0b48014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd219f176023d27c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b3a7fc640 1 -- 172.21.15.7:0/2684084988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b48011df0 con 0x55e72eb6de10 2024-03-19T01:24:16.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b3a7fc640 1 -- 172.21.15.7:0/2684084988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b48026070 con 0x55e72eb6de10 2024-03-19T01:24:16.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b3a7fc640 1 -- 172.21.15.7:0/2684084988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b48021660 con 0x55e72eb6de10 2024-03-19T01:24:16.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 -- 172.21.15.7:0/2684084988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6de10 msgr2=0x55e72eb6e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2684084988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6de10 0x55e72eb6e1f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0b48011dc0 tx=0x7f0b48014000 comp rx=0 tx=0).stop 2024-03-19T01:24:16.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 -- 172.21.15.7:0/2684084988 shutdown_connections 2024-03-19T01:24:16.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2684084988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72eb73ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2684084988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6e730 0x55e72eb73280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2684084988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6de10 0x55e72eb6e1f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 -- 172.21.15.7:0/2684084988 >> 172.21.15.7:0/2684084988 conn(0x55e72eb6f030 msgr2=0x55e72eb71420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 -- 172.21.15.7:0/2684084988 shutdown_connections 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.422+0000 7f0b5e049000 1 -- 172.21.15.7:0/2684084988 wait complete. 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 -- start start 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 Processor -- start 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72eb42510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72eb42a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72eb42f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e72e98e730 con 0x55e72eb6e730 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55e72e95a750 con 0x55e72eb6de10 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55e72eb75120 con 0x55e72eb737c0 2024-03-19T01:24:16.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 10 client.0 _ll_register_callbacks cb 0x55e72eb2e740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72eb42a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72eb42a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33946/0 (socket says 172.21.15.7:33946) 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b53fff640 1 -- 172.21.15.7:0/3913814858 learned_addr learned my addr 172.21.15.7:0/3913814858 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b58c63640 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72eb42510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b59464640 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72eb42f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b53fff640 1 -- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 msgr2=0x55e72eb42f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b53fff640 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72eb42f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b53fff640 1 -- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 msgr2=0x55e72eb42510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b53fff640 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72eb42510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b53fff640 1 -- 172.21.15.7:0/3913814858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e72e91a5b0 con 0x55e72eb6e730 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b58c63640 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72eb42510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:16.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b59464640 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72eb42f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b53fff640 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72eb42a50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0b440ae040 tx=0x7f0b4401ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13ef303528f7a74 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b337fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b337fe640 1 -- 172.21.15.7:0/3913814858 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b440bb020 con 0x55e72eb6e730 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b337fe640 1 -- 172.21.15.7:0/3913814858 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f0b440abb20 con 0x55e72eb6e730 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 -- 172.21.15.7:0/3913814858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b48026070 con 0x55e72eb6e730 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b337fe640 1 -- 172.21.15.7:0/3913814858 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0b440c5050 con 0x55e72eb6e730 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 10 client.5184 fetch_fsmap learned FSMap version 16 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 -- 172.21.15.7:0/3913814858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55e72ea138d0 con 0x55e72eb6e730 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b337fe640 1 -- 172.21.15.7:0/3913814858 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b48026070 con 0x55e72eb6e730 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b337fe640 1 -- 172.21.15.7:0/3913814858 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) ==== 2284+0+0 (secure 0 0 0) 0x7f0b440c8020 con 0x55e72eb6e730 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 10 client.5184 fetch_fsmap finished waiting for FSMap version 16 2024-03-19T01:24:16.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 10 client.5184 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 20 client.5184 populate_metadata read hostname 'smithi007' 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] conn(0x55e72eb86660 0x55e72eb88a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 4 client.5184 mds_command: new command op to 5001 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 -- 172.21.15.7:0/3913814858 --> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e72e92dbc0 con 0x55e72eb86660 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b58c63640 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] conn(0x55e72eb86660 0x55e72eb88a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b337fe640 10 client.5184 ms_handle_connect on v2:172.21.15.7:6834/4260183338 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b58c63640 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] conn(0x55e72eb86660 0x55e72eb88a50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fad646fba8474c47 server_cookie=175ea6f7b0cd8550 in_seq=0 out_seq=0 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b337fe640 1 -- 172.21.15.7:0/3913814858 <== mds.0 v2:172.21.15.7:6834/4260183338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e72e92dbc0 con 0x55e72eb86660 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b337fe640 10 client.5184 handle_command_reply: tid=0 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 client.5184 shutdown 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 -- 172.21.15.7:0/3913814858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 msgr2=0x55e72eb42a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72eb42a50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0b440ae040 tx=0x7f0b4401ab50 comp rx=0 tx=0).stop 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 -- 172.21.15.7:0/3913814858 shutdown_connections 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.426+0000 7f0b5e049000 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] conn(0x55e72eb86660 0x55e72eb88a50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.430+0000 7f0b5e049000 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72eb42f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.430+0000 7f0b337fe640 0 client.5184 ms_handle_reset on v2:172.21.15.7:6834/4260183338 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.430+0000 7f0b5e049000 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72eb42a50 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.430+0000 7f0b5e049000 1 --2- 172.21.15.7:0/3913814858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72eb42510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.430+0000 7f0b5e049000 1 -- 172.21.15.7:0/3913814858 >> 172.21.15.7:0/3913814858 conn(0x55e72eb6f030 msgr2=0x55e72eb71420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.430+0000 7f0b5e049000 1 -- 172.21.15.7:0/3913814858 shutdown_connections 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.430+0000 7f0b5e049000 1 -- 172.21.15.7:0/3913814858 wait complete. 2024-03-19T01:24:16.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.430+0000 7f0b5e049000 20 client.5184 trim_cache size 0 max 16384 2024-03-19T01:24:16.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b5e049000 1 Processor -- start 2024-03-19T01:24:16.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b5e049000 1 -- start start 2024-03-19T01:24:16.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72ebe2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72ebe2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72ebe3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b5e049000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541ec660 con 0x55e72eb6e730 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b5e049000 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541eb830 con 0x55e72eb6de10 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b5e049000 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55e72ea4dda0 con 0x55e72eb737c0 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72ebe2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72ebe2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33952/0 (socket says 172.21.15.7:33952) 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b53fff640 1 -- 172.21.15.7:0/1553778191 learned_addr learned my addr 172.21.15.7:0/1553778191 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b53fff640 1 -- 172.21.15.7:0/1553778191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 msgr2=0x55e72ebe3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b53fff640 1 --2- 172.21.15.7:0/1553778191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72ebe3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b53fff640 1 -- 172.21.15.7:0/1553778191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 msgr2=0x55e72ebe2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b53fff640 1 --2- 172.21.15.7:0/1553778191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72ebe2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b53fff640 1 -- 172.21.15.7:0/1553778191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e72e9ea300 con 0x55e72eb6e730 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b53fff640 1 --2- 172.21.15.7:0/1553778191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72ebe2c40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0b440097c0 tx=0x7f0b440b4000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfba3269f3a2469e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b39ffb640 1 -- 172.21.15.7:0/1553778191 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b440bb040 con 0x55e72eb6e730 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b39ffb640 1 -- 172.21.15.7:0/1553778191 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b44000e50 con 0x55e72eb6e730 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.446+0000 7f0b39ffb640 1 -- 172.21.15.7:0/1553778191 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b440b0040 con 0x55e72eb6e730 2024-03-19T01:24:16.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/1553778191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 msgr2=0x55e72ebe2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/1553778191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72ebe2c40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0b440097c0 tx=0x7f0b440b4000 comp rx=0 tx=0).stop 2024-03-19T01:24:16.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/1553778191 shutdown_connections 2024-03-19T01:24:16.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/1553778191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb737c0 0x55e72ebe3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/1553778191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb6e730 0x55e72ebe2c40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/1553778191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72ebe2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/1553778191 >> 172.21.15.7:0/1553778191 conn(0x55e72eb6f030 msgr2=0x55e72eb3ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:16.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/1553778191 shutdown_connections 2024-03-19T01:24:16.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/1553778191 wait complete. 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- start start 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 Processor -- start 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72ebb7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb6e730 0x55e72ebb79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb737c0 0x55e72ebb7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e72e9919a0 con 0x55e72eb737c0 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55e72e92a7b0 con 0x55e72eb6de10 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55e72e996cc0 con 0x55e72eb6e730 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 10 client.0 _ll_register_callbacks cb 0x7f0b440ae070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b59464640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb737c0 0x55e72ebb7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b59464640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb737c0 0x55e72ebb7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33954/0 (socket says 172.21.15.7:33954) 2024-03-19T01:24:16.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b59464640 1 -- 172.21.15.7:0/2402948811 learned_addr learned my addr 172.21.15.7:0/2402948811 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b58c63640 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72ebb7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b53fff640 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb6e730 0x55e72ebb79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b59464640 1 -- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb6e730 msgr2=0x55e72ebb79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b59464640 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb6e730 0x55e72ebb79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b59464640 1 -- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 msgr2=0x55e72ebb7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b59464640 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72ebb7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b59464640 1 -- 172.21.15.7:0/2402948811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e72e942190 con 0x55e72eb737c0 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b58c63640 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72ebb7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b53fff640 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb6e730 0x55e72ebb79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b59464640 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb737c0 0x55e72ebb7f10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c001ce0 tx=0x7f0b4c0563b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3ec3686823a664 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 1 -- 172.21.15.7:0/2402948811 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c055fd0 con 0x55e72eb737c0 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 1 -- 172.21.15.7:0/2402948811 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f0b4c05ddf0 con 0x55e72eb737c0 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/2402948811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b44000e50 con 0x55e72eb737c0 2024-03-19T01:24:16.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 1 -- 172.21.15.7:0/2402948811 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0b4c05e280 con 0x55e72eb737c0 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 10 client.5190 fetch_fsmap learned FSMap version 16 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/2402948811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55e72e9264b0 con 0x55e72eb737c0 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 1 -- 172.21.15.7:0/2402948811 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b44000e50 con 0x55e72eb737c0 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 1 -- 172.21.15.7:0/2402948811 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 16) ==== 2284+0+0 (secure 0 0 0) 0x7f0b4c004b70 con 0x55e72eb737c0 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 10 client.5190 fetch_fsmap finished waiting for FSMap version 16 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 10 client.5190 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 20 client.5190 populate_metadata read hostname 'smithi007' 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] conn(0x55e72ebe6780 0x55e72ebe6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 4 client.5190 mds_command: new command op to 5001 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/2402948811 --> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] -- command(tid 0: {"prefix": "status"}) -- 0x55e72e99fbd0 con 0x55e72ebe6780 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b58c63640 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] conn(0x55e72ebe6780 0x55e72ebe6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b58c63640 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] conn(0x55e72ebe6780 0x55e72ebe6b60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26f25899c564d41e server_cookie=44b684be42b0e600 in_seq=0 out_seq=0 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 10 client.5190 ms_handle_connect on v2:172.21.15.7:6834/4260183338 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 1 -- 172.21.15.7:0/2402948811 <== mds.0 v2:172.21.15.7:6834/4260183338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55e72e99fbd0 con 0x55e72ebe6780 2024-03-19T01:24:16.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 10 client.5190 handle_command_reply: tid=0 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 client.5190 shutdown 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/2402948811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb737c0 msgr2=0x55e72ebb7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb737c0 0x55e72ebb7f10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c001ce0 tx=0x7f0b4c0563b0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/2402948811 shutdown_connections 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.7:6834/4260183338,v1:172.21.15.7:6835/4260183338] conn(0x55e72ebe6780 0x55e72ebe6b60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e72eb6e730 0x55e72ebb79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e72eb737c0 0x55e72ebb7f10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 --2- 172.21.15.7:0/2402948811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e72eb6de10 0x55e72ebb7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b337fe640 0 client.5190 ms_handle_reset on v2:172.21.15.7:6834/4260183338 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/2402948811 >> 172.21.15.7:0/2402948811 conn(0x55e72eb6f030 msgr2=0x55e72eb8bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/2402948811 shutdown_connections 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.450+0000 7f0b5e049000 1 -- 172.21.15.7:0/2402948811 wait complete. 2024-03-19T01:24:16.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5e049000 20 client.5190 trim_cache size 0 max 16384 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "dafc8aaa-17d4-48ac-88de-2f029fdf817b", 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 5001, 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 5.5776456659999996, 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 16, 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 30, 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 30, 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 12.472011269999999 2024-03-19T01:24:16.461 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:16.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 -- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0b40067690 msgr2=0x7f0b40069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0b40067690 0x7f0b40069b50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009af0 tx=0x7f0b48008040 comp rx=0 tx=0).stop 2024-03-19T01:24:16.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 -- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54141710 msgr2=0x7f0b54147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54141710 0x7f0b54147d90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00caa0 tx=0x7f0b4c00cf60 comp rx=0 tx=0).stop 2024-03-19T01:24:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 -- 172.21.15.7:0/1168686435 shutdown_connections 2024-03-19T01:24:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0b40067690 0x7f0b40069b50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b540b9ed0 0x7f0b54142da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54141710 0x7f0b54147d90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 --2- 172.21.15.7:0/1168686435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b540b47b0 0x7f0b541451a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 -- 172.21.15.7:0/1168686435 >> 172.21.15.7:0/1168686435 conn(0x7f0b54072df0 msgr2=0x7f0b54067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 -- 172.21.15.7:0/1168686435 shutdown_connections 2024-03-19T01:24:16.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.454+0000 7f0b5aec7640 1 -- 172.21.15.7:0/1168686435 wait complete. 2024-03-19T01:24:16.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-19T01:24:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.582+0000 7f0f185fb640 1 Processor -- start 2024-03-19T01:24:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.582+0000 7f0f185fb640 1 -- start start 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f185fb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f10074390 0x7f0f10148b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f185fb640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0f1013f660 0x7f0f10149060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f185fb640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f10070d60 con 0x7f0f100b4ec0 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f185fb640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f0f10070fd0 con 0x7f0f1013f660 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f185fb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0f10074770 con 0x7f0f10074390 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f16397640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f10074390 0x7f0f10148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f16b98640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0f1013f660 0x7f0f10149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f16b98640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f0f1013f660 0x7f0f10149060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:55812/0 (socket says 172.21.15.7:55812) 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f16b98640 1 -- 172.21.15.7:0/142682639 learned_addr learned my addr 172.21.15.7:0/142682639 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f15b96640 1 --1- 172.21.15.7:0/142682639 >> v1:172.21.15.167:6789/0 conn(0x7f0f100b4ec0 0x7f0f1013ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:58604/0 (socket says 172.21.15.7:58604) 2024-03-19T01:24:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f16b98640 1 -- 172.21.15.7:0/142682639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f10074390 msgr2=0x7f0f10148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f16b98640 1 --2- 172.21.15.7:0/142682639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f10074390 0x7f0f10148b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f16b98640 1 -- 172.21.15.7:0/142682639 >> v1:172.21.15.167:6789/0 conn(0x7f0f100b4ec0 legacy=0x7f0f1013ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f16b98640 1 -- 172.21.15.7:0/142682639 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1006fcc0 con 0x7f0f1013f660 2024-03-19T01:24:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f16b98640 1 --2- 172.21.15.7:0/142682639 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f1013f660 0x7f0f10149060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009e50 tx=0x7f0f0001c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=708b3033c1e846b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f15395640 1 -- 172.21.15.7:0/142682639 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f000a5770 con 0x7f0f1013f660 2024-03-19T01:24:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f15395640 1 -- 172.21.15.7:0/142682639 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f000a5910 con 0x7f0f1013f660 2024-03-19T01:24:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.586+0000 7f0f15395640 1 -- 172.21.15.7:0/142682639 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f000a5ce0 con 0x7f0f1013f660 2024-03-19T01:24:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- 172.21.15.7:0/142682639 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f1013f660 msgr2=0x7f0f10149060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 --2- 172.21.15.7:0/142682639 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f1013f660 0x7f0f10149060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009e50 tx=0x7f0f0001c9a0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- 172.21.15.7:0/142682639 shutdown_connections 2024-03-19T01:24:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 --2- 172.21.15.7:0/142682639 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f1013f660 0x7f0f10149060 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 --2- 172.21.15.7:0/142682639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f10074390 0x7f0f10148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- 172.21.15.7:0/142682639 >> 172.21.15.7:0/142682639 conn(0x7f0f100b5bf0 msgr2=0x7f0f100b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- 172.21.15.7:0/142682639 shutdown_connections 2024-03-19T01:24:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- 172.21.15.7:0/142682639 wait complete. 2024-03-19T01:24:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 Processor -- start 2024-03-19T01:24:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- start start 2024-03-19T01:24:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f10074390 0x7f0f10139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f100b4ec0 0x7f0f10142780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f1013f660 0x7f0f101e0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0f101526a0 con 0x7f0f1013f660 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0f10152520 con 0x7f0f10074390 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0f10152820 con 0x7f0f100b4ec0 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16b98640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f1013f660 0x7f0f101e0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16b98640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f1013f660 0x7f0f101e0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33970/0 (socket says 172.21.15.7:33970) 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16b98640 1 -- 172.21.15.7:0/3559727512 learned_addr learned my addr 172.21.15.7:0/3559727512 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16397640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f10074390 0x7f0f10139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f15b96640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f100b4ec0 0x7f0f10142780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16b98640 1 -- 172.21.15.7:0/3559727512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f100b4ec0 msgr2=0x7f0f10142780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16b98640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f100b4ec0 0x7f0f10142780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16b98640 1 -- 172.21.15.7:0/3559727512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f10074390 msgr2=0x7f0f10139a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16b98640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f10074390 0x7f0f10139a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16b98640 1 -- 172.21.15.7:0/3559727512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1013b6d0 con 0x7f0f1013f660 2024-03-19T01:24:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16397640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f10074390 0x7f0f10139a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f16b98640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f1013f660 0x7f0f101e0df0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0f000af040 tx=0x7f0f00019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f0effd640 1 -- 172.21.15.7:0/3559727512 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f000bc020 con 0x7f0f1013f660 2024-03-19T01:24:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f0effd640 1 -- 172.21.15.7:0/3559727512 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f000b8670 con 0x7f0f1013f660 2024-03-19T01:24:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f0effd640 1 -- 172.21.15.7:0/3559727512 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f000b8a40 con 0x7f0f1013f660 2024-03-19T01:24:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1013ce50 con 0x7f0f1013f660 2024-03-19T01:24:16.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.590+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f10141960 con 0x7f0f1013f660 2024-03-19T01:24:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.594+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f10075460 con 0x7f0f1013f660 2024-03-19T01:24:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.594+0000 7f0f0effd640 1 -- 172.21.15.7:0/3559727512 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0f000bf020 con 0x7f0f1013f660 2024-03-19T01:24:16.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.594+0000 7f0f0effd640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0efc067660 0x7f0efc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.594+0000 7f0f0effd640 1 -- 172.21.15.7:0/3559727512 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f0f001207f0 con 0x7f0f1013f660 2024-03-19T01:24:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.594+0000 7f0f16397640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0efc067660 0x7f0efc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:16.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.594+0000 7f0f16397640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0efc067660 0x7f0efc069b20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009b10 tx=0x7f0f08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:16.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.598+0000 7f0f0effd640 1 -- 172.21.15.7:0/3559727512 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0f000c6030 con 0x7f0f1013f660 2024-03-19T01:24:16.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0f100b80d0 con 0x7f0f1013f660 2024-03-19T01:24:16.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f0effd640 1 -- 172.21.15.7:0/3559727512 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f0f00016620 con 0x7f0f1013f660 2024-03-19T01:24:16.797 INFO:teuthology.orchestra.run.smithi007.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0efc067660 msgr2=0x7f0efc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0efc067660 0x7f0efc069b20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009b10 tx=0x7f0f08008040 comp rx=0 tx=0).stop 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f1013f660 msgr2=0x7f0f101e0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f1013f660 0x7f0f101e0df0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0f000af040 tx=0x7f0f00019740 comp rx=0 tx=0).stop 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 shutdown_connections 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0efc067660 0x7f0efc069b20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f100b4ec0 0x7f0f10142780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f1013f660 0x7f0f101e0df0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 --2- 172.21.15.7:0/3559727512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f10074390 0x7f0f10139a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 >> 172.21.15.7:0/3559727512 conn(0x7f0f100b5bf0 msgr2=0x7f0f10148b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 shutdown_connections 2024-03-19T01:24:16.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:16.794+0000 7f0f185fb640 1 -- 172.21.15.7:0/3559727512 wait complete. 2024-03-19T01:24:16.819 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-19T01:24:16.819 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T01:24:16.819 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi007.front.sepia.ceph.com 2024-03-19T01:24:16.819 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-19T01:24:16.819 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T01:24:16.819 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-19T01:24:16.819 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T01:24:16.819 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T01:24:16.820 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-19T01:24:16.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:16.820 DEBUG:teuthology.orchestra.run.smithi007:> ip addr 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2024-03-19T01:24:16.826 INFO:teuthology.orchestra.run.smithi007.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 42455sec preferred_lft 42455sec 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-19T01:24:16.827 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 02:42:33:01:ef:e0 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:24:16.828 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-19T01:24:16.828 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:24:16.828 INFO:teuthology.orchestra.run.smithi007.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-19T01:24:16.828 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-19T01:24:16.828 INFO:teuthology.orchestra.run.smithi007.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-19T01:24:16.828 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:24:16.828 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-19T01:24:16.829 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:24:16.829 INFO:teuthology.orchestra.run.smithi007.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-19T01:24:16.829 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:24:16.829 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-19T01:24:16.829 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:24:16.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:16.829 DEBUG:teuthology.orchestra.run.smithi007:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T01:24:16.877 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-03-19T01:24:16.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:16.879 DEBUG:teuthology.orchestra.run.smithi007:> route 2024-03-19T01:24:16.925 INFO:teuthology.orchestra.run.smithi007.stdout:Kernel IP routing table 2024-03-19T01:24:16.925 INFO:teuthology.orchestra.run.smithi007.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T01:24:16.925 INFO:teuthology.orchestra.run.smithi007.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-19T01:24:16.925 INFO:teuthology.orchestra.run.smithi007.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-19T01:24:16.925 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-19T01:24:16.925 INFO:teuthology.orchestra.run.smithi007.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T01:24:16.926 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:24:16.926 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-19T01:24:16.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-19T01:24:16.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-19T01:24:16.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-19T01:24:16.926 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-19T01:24:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:24:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:24:17.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:17.043 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-03-19T01:24:17.095 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-19T01:24:17.096 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-19T01:24:17.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:17.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set brx.0 up 2024-03-19T01:24:17.152 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-19T01:24:17.152 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T01:24:17.152 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-19T01:24:17.201 INFO:teuthology.orchestra.run.smithi007.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-19T01:24:17.201 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T01:24:17.201 DEBUG:teuthology.orchestra.run.smithi007:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-19T01:24:17.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-03-19T01:24:17.306 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T01:24:17.353 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-03-19T01:24:17.353 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-03-19T01:24:17.353 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-03-19T01:24:17.353 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-03-19T01:24:17.353 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-03-19T01:24:17.354 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-03-19T01:24:17.354 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-03-19T01:24:17.354 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-03-19T01:24:17.354 INFO:teuthology.orchestra.run.smithi007.stdout:/run/lock 2024-03-19T01:24:17.354 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-03-19T01:24:17.354 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-03-19T01:24:17.355 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-03-19T01:24:17.355 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T01:24:17.355 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-03-19T01:24:17.355 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-03-19T01:24:17.355 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-03-19T01:24:17.355 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-03-19T01:24:17.355 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/fs/nfsd 2024-03-19T01:24:17.356 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-03-19T01:24:17.356 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-03-19T01:24:17.356 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2024-03-19T01:24:17.356 INFO:teuthology.orchestra.run.smithi007.stdout:/run/rpc_pipefs 2024-03-19T01:24:17.356 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-03-19T01:24:17.356 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-03-19T01:24:17.356 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-0 2024-03-19T01:24:17.356 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-1 2024-03-19T01:24:17.357 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-2 2024-03-19T01:24:17.357 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-3 2024-03-19T01:24:17.357 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-03-19T01:24:17.357 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:24:17.357 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:24:17.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:17.359 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-03-19T01:24:17.407 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-19T01:24:17.408 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-19T01:24:17.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-03-19T01:24:17.467 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T01:24:17.515 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-03-19T01:24:17.515 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-03-19T01:24:17.515 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-03-19T01:24:17.515 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-03-19T01:24:17.515 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/run/lock 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/fs/nfsd 2024-03-19T01:24:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/run/rpc_pipefs 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-0 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-1 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-2 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-3 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:24:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:24:17.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:17.518 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-03-19T01:24:17.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:2024-03-19T01:24:17.526+0000 7f33f65ae580 -1 init, newargv = 0x557a68120b90 newargc=15 2024-03-19T01:24:17.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:ceph-fuse[36292]: starting ceph client 2024-03-19T01:24:17.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:ceph-fuse[36292]: starting fuse 2024-03-19T01:24:18.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-03-19T01:24:18.649 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/run/lock 2024-03-19T01:24:18.697 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/fs/nfsd 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/run/rpc_pipefs 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-03-19T01:24:18.698 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-03-19T01:24:18.699 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-0 2024-03-19T01:24:18.699 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-1 2024-03-19T01:24:18.699 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-2 2024-03-19T01:24:18.699 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph/osd/ceph-3 2024-03-19T01:24:18.699 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-03-19T01:24:18.699 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:24:18.699 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:24:18.699 INFO:teuthology.orchestra.run.smithi007.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-19T01:24:18.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:18.700 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-03-19T01:24:18.747 INFO:teuthology.orchestra.run.smithi007.stdout:45 2024-03-19T01:24:18.748 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo stdin-killer -- python3 -c ' 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> import glob 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> import re 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> import os 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> import subprocess 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> def _find_admin_socket(client_name): 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> files = glob.glob(asok_path) 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-19T01:24:18.748 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> # Given a non-glob path, it better be there 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> if "*" not in asok_path: 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> assert(len(files) == 1) 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> return files[0] 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> for f in files: 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> contents = proc_f.read() 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> if mountpoint in contents: 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> return f 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> print(_find_admin_socket("client.0")) 2024-03-19T01:24:18.749 DEBUG:teuthology.orchestra.run.smithi007:> ' 2024-03-19T01:24:18.872 INFO:teuthology.orchestra.run.smithi007.stdout:/var/run/ceph/ceph-client.0.36292.asok 2024-03-19T01:24:18.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:24:18.884 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.36292.asok 2024-03-19T01:24:18.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:18.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.36292.asok status 2024-03-19T01:24:19.037 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:19.037 INFO:teuthology.orchestra.run.smithi007.stdout: "metadata": { 2024-03-19T01:24:19.037 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-19T01:24:19.037 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T01:24:19.037 INFO:teuthology.orchestra.run.smithi007.stdout: "entity_id": "0", 2024-03-19T01:24:19.037 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": "smithi007", 2024-03-19T01:24:19.037 INFO:teuthology.orchestra.run.smithi007.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "pid": "36292", 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "root": "/" 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_count": 0, 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_pinned_count": 0, 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 5199, 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "inst": { 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "name": { 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "client", 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "num": 5199 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-03-19T01:24:19.038 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1284733917 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1284733917 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "inst_str": "client.5199 192.168.0.1:0/1284733917", 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "addr_str": "192.168.0.1:0/1284733917", 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "inode_count": 1, 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_epoch": 16, 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch": 30, 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch_barrier": 0, 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklisted": false, 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs" 2024-03-19T01:24:19.039 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:19.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:19.051 DEBUG:teuthology.orchestra.run.smithi007:> stat --file-system '--printf=%T 2024-03-19T01:24:19.051 DEBUG:teuthology.orchestra.run.smithi007:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-19T01:24:19.057 INFO:teuthology.orchestra.run.smithi007.stdout:fuseblk 2024-03-19T01:24:19.057 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-19T01:24:19.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:19.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-19T01:24:19.115 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-19T01:24:19.167 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.CPK8WGwFfJ 2024-03-19T01:24:19.168 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:24:19.168 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/tmp/tmp.CPK8WGwFfJ 2024-03-19T01:24:19.216 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p dir1/dir2/dir3') 2024-03-19T01:24:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T01:24:19.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:24:19.375 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'chmod 755 /tmp/test-cephfsmp5s52lw') 2024-03-19T01:24:19.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T01:24:19.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:24:19.424 INFO:tasks.cephfs.test_cephfs_shell:Running script "/tmp/test-cephfsmp5s52lw" 2024-03-19T01:24:19.424 DEBUG:teuthology.orchestra.run.smithi007:> cephfs-shell -b /tmp/test-cephfsmp5s52lw 2024-03-19T01:24:19.795 INFO:teuthology.orchestra.run.smithi007.stderr:Unrecognized command 2024-03-19T01:24:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 127 2024-03-19T01:24:19.823 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-19T01:24:19.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-19T01:24:19.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-19T01:24:19.898 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-19T01:24:19.898 INFO:tasks.cephfs.mount:Terminating background process 2024-03-19T01:24:19.899 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:24:19.899 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:24:19.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:20.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.066+0000 7f2b079b5640 1 Processor -- start 2024-03-19T01:24:20.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.066+0000 7f2b079b5640 1 -- start start 2024-03-19T01:24:20.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b079b5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2b00147c10 0x7f2b001480a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b079b5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b0014fe20 0x7f2b00152210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b079b5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b0009f440 con 0x7f2b0009f060 2024-03-19T01:24:20.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b079b5640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f2b000ae9c0 con 0x7f2b00147c10 2024-03-19T01:24:20.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b079b5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2b000aeca0 con 0x7f2b0014fe20 2024-03-19T01:24:20.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b05751640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2b0009f060 0x7f2b0006dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37686/0 (socket says 172.21.15.7:37686) 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b05751640 1 -- 172.21.15.7:0/3072898640 learned_addr learned my addr 172.21.15.7:0/3072898640 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b04f50640 1 --2- 172.21.15.7:0/3072898640 >> v2:172.21.15.167:3300/0 conn(0x7f2b00147c10 0x7f2b001480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2300068324 0 0) 0x7f2b0009f440 con 0x7f2b0009f060 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b05f52640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b0014fe20 0x7f2b00152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ae8003530 con 0x7f2b0009f060 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3765714724 0 0) 0x7f2ae8003530 con 0x7f2b0009f060 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b0014fe20 msgr2=0x7f2b00152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b0014fe20 0x7f2b00152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 >> v2:172.21.15.167:3300/0 conn(0x7f2b00147c10 msgr2=0x7f2b001480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 --2- 172.21.15.7:0/3072898640 >> v2:172.21.15.167:3300/0 conn(0x7f2b00147c10 0x7f2b001480a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0006d180 con 0x7f2b0009f060 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f2af8001a00 con 0x7f2b0009f060 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 >> v1:172.21.15.7:6789/0 conn(0x7f2b0009f060 legacy=0x7f2b0006dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b0014fe20 0x7f2ae80039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2ae8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ae8004d50 0x7f2ae80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2b000ae9c0 con 0x7f2ae8003f40 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2b000aeca0 con 0x7f2ae8004d50 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae800bab0 con 0x7f2b0014fe20 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b04f50640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2ae8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b05751640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b0014fe20 0x7f2ae80039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b05f52640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ae8004d50 0x7f2ae80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b04f50640 1 -- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b0014fe20 msgr2=0x7f2ae80039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b04f50640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b0014fe20 0x7f2ae80039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b04f50640 1 -- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ae8004d50 msgr2=0x7f2ae80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b04f50640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ae8004d50 0x7f2ae80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b04f50640 1 -- 172.21.15.7:0/3072898640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae8003530 con 0x7f2ae8003f40 2024-03-19T01:24:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.070+0000 7f2b05f52640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ae8004d50 0x7f2ae80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b04f50640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2ae8004810 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009ae0 tx=0x7f2af001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea61354e7e6456be server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af00096b0 con 0x7f2ae8003f40 2024-03-19T01:24:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2af00a6490 con 0x7f2ae8003f40 2024-03-19T01:24:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2afffff640 1 -- 172.21.15.7:0/3072898640 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af00a6880 con 0x7f2ae8003f40 2024-03-19T01:24:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 -- 172.21.15.7:0/3072898640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 msgr2=0x7f2ae8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 --2- 172.21.15.7:0/3072898640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2ae8004810 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009ae0 tx=0x7f2af001cab0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b05f52640 1 -- 172.21.15.7:0/3072898640 reap_dead start 2024-03-19T01:24:20.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 -- 172.21.15.7:0/3072898640 shutdown_connections 2024-03-19T01:24:20.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 -- 172.21.15.7:0/3072898640 >> 172.21.15.7:0/3072898640 conn(0x7f2b000b6510 msgr2=0x7f2b000b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:20.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 -- 172.21.15.7:0/3072898640 shutdown_connections 2024-03-19T01:24:20.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 -- 172.21.15.7:0/3072898640 wait complete. 2024-03-19T01:24:20.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 Processor -- start 2024-03-19T01:24:20.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 -- start start 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2b000b5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b00071f30 0x7f2b0013df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b001dfd80 0x7f2b0013bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2b000b0bf0 con 0x7f2ae8003f40 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2b000b0a70 con 0x7f2b001dfd80 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b079b5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0014ab00 con 0x7f2b00071f30 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b05751640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2b000b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b05751640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2b000b5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34010/0 (socket says 172.21.15.7:34010) 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.074+0000 7f2b05751640 1 -- 172.21.15.7:0/4139184829 learned_addr learned my addr 172.21.15.7:0/4139184829 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b05f52640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b001dfd80 0x7f2b0013bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b04f50640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b00071f30 0x7f2b0013df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b05751640 1 -- 172.21.15.7:0/4139184829 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b00071f30 msgr2=0x7f2b0013df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b05751640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b00071f30 0x7f2b0013df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b05751640 1 -- 172.21.15.7:0/4139184829 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b001dfd80 msgr2=0x7f2b0013bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b05751640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b001dfd80 0x7f2b0013bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b05751640 1 -- 172.21.15.7:0/4139184829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b000b4b30 con 0x7f2ae8003f40 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b05751640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2b000b5730 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2af8005fd0 tx=0x7f2af800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2afdffb640 1 -- 172.21.15.7:0/4139184829 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af800ee50 con 0x7f2ae8003f40 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2afdffb640 1 -- 172.21.15.7:0/4139184829 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2af8018070 con 0x7f2ae8003f40 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2afdffb640 1 -- 172.21.15.7:0/4139184829 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8013660 con 0x7f2ae8003f40 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b000b4f50 con 0x7f2ae8003f40 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b00071740 con 0x7f2ae8003f40 2024-03-19T01:24:20.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b00075460 con 0x7f2ae8003f40 2024-03-19T01:24:20.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2afdffb640 1 -- 172.21.15.7:0/4139184829 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2af8013800 con 0x7f2ae8003f40 2024-03-19T01:24:20.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2afdffb640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2aec067690 0x7f2aec069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2afdffb640 1 -- 172.21.15.7:0/4139184829 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f2af8086810 con 0x7f2ae8003f40 2024-03-19T01:24:20.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b04f50640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2aec067690 0x7f2aec069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.078+0000 7f2b04f50640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2aec067690 0x7f2aec069b50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2af00ad000 tx=0x7f2af00b2040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:20.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.082+0000 7f2afdffb640 1 -- 172.21.15.7:0/4139184829 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2af8057600 con 0x7f2ae8003f40 2024-03-19T01:24:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.242+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b000b97a0 con 0x7f2ae8003f40 2024-03-19T01:24:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2afdffb640 1 -- 172.21.15.7:0/4139184829 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16633 (secure 0 0 0) 0x7f2af8056f00 con 0x7f2ae8003f40 2024-03-19T01:24:20.248 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:20.248 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":30,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:24:11.876754+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:24:08.795930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-19T01:24:09.233323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2aec067690 msgr2=0x7f2aec069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2aec067690 0x7f2aec069b50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2af00ad000 tx=0x7f2af00b2040 comp rx=0 tx=0).stop 2024-03-19T01:24:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 msgr2=0x7f2b000b5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2b000b5730 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2af8005fd0 tx=0x7f2af800e530 comp rx=0 tx=0).stop 2024-03-19T01:24:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 shutdown_connections 2024-03-19T01:24:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2aec067690 0x7f2aec069b50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b00071f30 0x7f2b0013df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b001dfd80 0x7f2b0013bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 --2- 172.21.15.7:0/4139184829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ae8003f40 0x7f2b000b5730 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 >> 172.21.15.7:0/4139184829 conn(0x7f2b000b6510 msgr2=0x7f2b0013ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 shutdown_connections 2024-03-19T01:24:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.246+0000 7f2b079b5640 1 -- 172.21.15.7:0/4139184829 wait complete. 2024-03-19T01:24:20.265 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:24:08.795930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:24:09.233323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:24:20.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:24:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.390+0000 7f1a9f87e640 1 Processor -- start 2024-03-19T01:24:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.390+0000 7f1a9f87e640 1 -- start start 2024-03-19T01:24:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9f87e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f1a98072810 0x7f1a9813f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9f87e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1a980b4830 0x7f1a9813f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9f87e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 0x7f1a98145760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9f87e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f1a980aefe0 con 0x7f1a980b4830 2024-03-19T01:24:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9f87e640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f1a980b5920 con 0x7f1a98072810 2024-03-19T01:24:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9f87e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a980af230 con 0x7f1a981452d0 2024-03-19T01:24:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9ce19640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1a980b4830 0x7f1a9813f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9ce19640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1a980b4830 0x7f1a9813f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34022/0 (socket says 172.21.15.7:34022) 2024-03-19T01:24:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9ce19640 1 -- 172.21.15.7:0/2018059962 learned_addr learned my addr 172.21.15.7:0/2018059962 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9de1b640 1 --2- 172.21.15.7:0/2018059962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 0x7f1a98145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9d61a640 1 --2- 172.21.15.7:0/2018059962 >> v2:172.21.15.167:3300/0 conn(0x7f1a98072810 0x7f1a9813f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9ce19640 1 -- 172.21.15.7:0/2018059962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 msgr2=0x7f1a98145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9ce19640 1 --2- 172.21.15.7:0/2018059962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 0x7f1a98145760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9ce19640 1 -- 172.21.15.7:0/2018059962 >> v2:172.21.15.167:3300/0 conn(0x7f1a98072810 msgr2=0x7f1a9813f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9ce19640 1 --2- 172.21.15.7:0/2018059962 >> v2:172.21.15.167:3300/0 conn(0x7f1a98072810 0x7f1a9813f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9ce19640 1 -- 172.21.15.7:0/2018059962 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a98145ca0 con 0x7f1a980b4830 2024-03-19T01:24:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.394+0000 7f1a9ce19640 1 --2- 172.21.15.7:0/2018059962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 0x7f1a9813f540 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1a98072ed0 tx=0x7f1a8c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde32f6444ef4295 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a97fff640 1 -- 172.21.15.7:0/2018059962 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0ab650 con 0x7f1a980b4830 2024-03-19T01:24:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a97fff640 1 -- 172.21.15.7:0/2018059962 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a8c0b0070 con 0x7f1a980b4830 2024-03-19T01:24:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a97fff640 1 -- 172.21.15.7:0/2018059962 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0aba10 con 0x7f1a980b4830 2024-03-19T01:24:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- 172.21.15.7:0/2018059962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 msgr2=0x7f1a9813f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/2018059962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 0x7f1a9813f540 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1a98072ed0 tx=0x7f1a8c01c490 comp rx=0 tx=0).stop 2024-03-19T01:24:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- 172.21.15.7:0/2018059962 shutdown_connections 2024-03-19T01:24:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/2018059962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 0x7f1a98145760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/2018059962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 0x7f1a9813f540 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/2018059962 >> v2:172.21.15.167:3300/0 conn(0x7f1a98072810 0x7f1a9813f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- 172.21.15.7:0/2018059962 >> 172.21.15.7:0/2018059962 conn(0x7f1a9806e1b0 msgr2=0x7f1a9806e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- 172.21.15.7:0/2018059962 shutdown_connections 2024-03-19T01:24:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- 172.21.15.7:0/2018059962 wait complete. 2024-03-19T01:24:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 Processor -- start 2024-03-19T01:24:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- start start 2024-03-19T01:24:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a98072810 0x7f1a980b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 0x7f1a98070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 0x7f1a98071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9806bad0 con 0x7f1a980b4830 2024-03-19T01:24:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a980b45f0 con 0x7f1a98072810 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9806bc50 con 0x7f1a981452d0 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9ce19640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 0x7f1a98070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9ce19640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 0x7f1a98070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34030/0 (socket says 172.21.15.7:34030) 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9ce19640 1 -- 172.21.15.7:0/515081018 learned_addr learned my addr 172.21.15.7:0/515081018 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9d61a640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a98072810 0x7f1a980b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9de1b640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 0x7f1a98071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9ce19640 1 -- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 msgr2=0x7f1a98071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9ce19640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 0x7f1a98071380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9ce19640 1 -- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a98072810 msgr2=0x7f1a980b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9ce19640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a98072810 0x7f1a980b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9ce19640 1 -- 172.21.15.7:0/515081018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a98146d00 con 0x7f1a980b4830 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9d61a640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a98072810 0x7f1a980b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9de1b640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 0x7f1a98071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9ce19640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 0x7f1a98070e40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c01ab40 tx=0x7f1a8c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a95ffb640 1 -- 172.21.15.7:0/515081018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0b4800 con 0x7f1a980b4830 2024-03-19T01:24:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a95ffb640 1 -- 172.21.15.7:0/515081018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a8c0b49a0 con 0x7f1a980b4830 2024-03-19T01:24:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a9809f180 con 0x7f1a980b4830 2024-03-19T01:24:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a95ffb640 1 -- 172.21.15.7:0/515081018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0b4d50 con 0x7f1a980b4830 2024-03-19T01:24:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.398+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a981492c0 con 0x7f1a980b4830 2024-03-19T01:24:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.402+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a98075570 con 0x7f1a980b4830 2024-03-19T01:24:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.402+0000 7f1a95ffb640 1 -- 172.21.15.7:0/515081018 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1a8c0c7440 con 0x7f1a980b4830 2024-03-19T01:24:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.402+0000 7f1a95ffb640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a84067660 0x7f1a84069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.402+0000 7f1a95ffb640 1 -- 172.21.15.7:0/515081018 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f1a8c1206b0 con 0x7f1a980b4830 2024-03-19T01:24:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.402+0000 7f1a9d61a640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a84067660 0x7f1a84069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.402+0000 7f1a9d61a640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a84067660 0x7f1a84069b20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a900097c0 tx=0x7f1a90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.406+0000 7f1a95ffb640 1 -- 172.21.15.7:0/515081018 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1a8c016110 con 0x7f1a980b4830 2024-03-19T01:24:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.558+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1a9806ca90 con 0x7f1a980b4830 2024-03-19T01:24:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a95ffb640 1 -- 172.21.15.7:0/515081018 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7f1a8c0f13c0 con 0x7f1a980b4830 2024-03-19T01:24:20.564 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:24:20.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a84067660 msgr2=0x7f1a84069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a84067660 0x7f1a84069b20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a900097c0 tx=0x7f1a90008040 comp rx=0 tx=0).stop 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 msgr2=0x7f1a98070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 0x7f1a98070e40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c01ab40 tx=0x7f1a8c01acd0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 shutdown_connections 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1a84067660 0x7f1a84069b20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a981452d0 0x7f1a98071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1a980b4830 0x7f1a98070e40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 --2- 172.21.15.7:0/515081018 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a98072810 0x7f1a980b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 >> 172.21.15.7:0/515081018 conn(0x7f1a9806e1b0 msgr2=0x7f1a9806e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 shutdown_connections 2024-03-19T01:24:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.562+0000 7f1a9f87e640 1 -- 172.21.15.7:0/515081018 wait complete. 2024-03-19T01:24:20.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:24:20.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c7774640 1 Processor -- start 2024-03-19T01:24:20.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c7774640 1 -- start start 2024-03-19T01:24:20.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c7774640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f63c00b9bb0 0x7f63c00afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c7774640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c013dff0 0x7f63c013e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c7774640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c00b9f90 con 0x7f63c00af4f0 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c7774640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f63c00b5e30 con 0x7f63c00b9bb0 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c7774640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f63c00b62c0 con 0x7f63c013dff0 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c5510640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f63c00b9bb0 0x7f63c00afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c5510640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f63c00b9bb0 0x7f63c00afb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34042/0 (socket says 172.21.15.7:34042) 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c5510640 1 -- 172.21.15.7:0/736571664 learned_addr learned my addr 172.21.15.7:0/736571664 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c5d11640 1 --2- 172.21.15.7:0/736571664 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c013dff0 0x7f63c013e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c5510640 1 -- 172.21.15.7:0/736571664 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c013dff0 msgr2=0x7f63c013e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c5510640 1 --2- 172.21.15.7:0/736571664 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c013dff0 0x7f63c013e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c5510640 1 -- 172.21.15.7:0/736571664 >> v1:172.21.15.167:6789/0 conn(0x7f63c00af4f0 legacy=0x7f63c0147670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c5510640 1 -- 172.21.15.7:0/736571664 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c00b5ac0 con 0x7f63c00b9bb0 2024-03-19T01:24:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.714+0000 7f63c5d11640 1 --2- 172.21.15.7:0/736571664 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c013dff0 0x7f63c013e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5510640 1 --2- 172.21.15.7:0/736571664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00b9bb0 0x7f63c00afb30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009870 tx=0x7f63b801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f1532f835819c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63b7fff640 1 -- 172.21.15.7:0/736571664 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b80b0070 con 0x7f63c00b9bb0 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63b7fff640 1 -- 172.21.15.7:0/736571664 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b80ab600 con 0x7f63c00b9bb0 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63b7fff640 1 -- 172.21.15.7:0/736571664 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b80ab9d0 con 0x7f63c00b9bb0 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 -- 172.21.15.7:0/736571664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00b9bb0 msgr2=0x7f63c00afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 --2- 172.21.15.7:0/736571664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00b9bb0 0x7f63c00afb30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009870 tx=0x7f63b801c400 comp rx=0 tx=0).stop 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 -- 172.21.15.7:0/736571664 shutdown_connections 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 --2- 172.21.15.7:0/736571664 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c013dff0 0x7f63c013e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 --2- 172.21.15.7:0/736571664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c00b9bb0 0x7f63c00afb30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 -- 172.21.15.7:0/736571664 >> 172.21.15.7:0/736571664 conn(0x7f63c006e1b0 msgr2=0x7f63c006e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:20.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 -- 172.21.15.7:0/736571664 shutdown_connections 2024-03-19T01:24:20.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 -- 172.21.15.7:0/736571664 wait complete. 2024-03-19T01:24:20.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 Processor -- start 2024-03-19T01:24:20.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 -- start start 2024-03-19T01:24:20.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c00af4f0 0x7f63c0144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c00b9bb0 0x7f63c0143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c013dff0 0x7f63c01de2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f63c006be60 con 0x7f63c013dff0 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f63c006bce0 con 0x7f63c00b9bb0 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c7774640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f63c006bfe0 con 0x7f63c00af4f0 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5d11640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c013dff0 0x7f63c01de2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5d11640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c013dff0 0x7f63c01de2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34050/0 (socket says 172.21.15.7:34050) 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5d11640 1 -- 172.21.15.7:0/1542806258 learned_addr learned my addr 172.21.15.7:0/1542806258 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c4d0f640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c00b9bb0 0x7f63c0143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5d11640 1 -- 172.21.15.7:0/1542806258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c00af4f0 msgr2=0x7f63c0144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5510640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c00af4f0 0x7f63c0144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5d11640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c00af4f0 0x7f63c0144a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5d11640 1 -- 172.21.15.7:0/1542806258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c00b9bb0 msgr2=0x7f63c0143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5d11640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c00b9bb0 0x7f63c0143850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5d11640 1 -- 172.21.15.7:0/1542806258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c0143d90 con 0x7f63c013dff0 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c4d0f640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c00b9bb0 0x7f63c0143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:20.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.718+0000 7f63c5d11640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c013dff0 0x7f63c01de2a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00cef0 tx=0x7f63bc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63b5ffb640 1 -- 172.21.15.7:0/1542806258 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc01a070 con 0x7f63c013dff0 2024-03-19T01:24:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63b5ffb640 1 -- 172.21.15.7:0/1542806258 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63bc00d860 con 0x7f63c013dff0 2024-03-19T01:24:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c014f480 con 0x7f63c013dff0 2024-03-19T01:24:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63b5ffb640 1 -- 172.21.15.7:0/1542806258 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc00dc50 con 0x7f63c013dff0 2024-03-19T01:24:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c0144010 con 0x7f63c013dff0 2024-03-19T01:24:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c01ddf90 con 0x7f63c013dff0 2024-03-19T01:24:20.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63b5ffb640 1 -- 172.21.15.7:0/1542806258 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f63bc01d070 con 0x7f63c013dff0 2024-03-19T01:24:20.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63b5ffb640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f63a4067660 0x7f63a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:20.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63c5510640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f63a4067660 0x7f63a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:20.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63b5ffb640 1 -- 172.21.15.7:0/1542806258 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f63bc086490 con 0x7f63c013dff0 2024-03-19T01:24:20.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.722+0000 7f63c5510640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f63a4067660 0x7f63a4069b20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63b80ad000 tx=0x7f63b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:20.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.726+0000 7f63b5ffb640 1 -- 172.21.15.7:0/1542806258 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f63bc024020 con 0x7f63c013dff0 2024-03-19T01:24:20.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.890+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f63c0147450 con 0x7f63c013dff0 2024-03-19T01:24:20.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.890+0000 7f63b5ffb640 1 -- 172.21.15.7:0/1542806258 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) ==== 104+0+11 (secure 0 0 0) 0x7f63c0143d90 con 0x7f63c013dff0 2024-03-19T01:24:20.894 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:24:20.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f63a4067660 msgr2=0x7f63a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f63a4067660 0x7f63a4069b20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63b80ad000 tx=0x7f63b8008040 comp rx=0 tx=0).stop 2024-03-19T01:24:20.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c013dff0 msgr2=0x7f63c01de2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:20.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c013dff0 0x7f63c01de2a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00cef0 tx=0x7f63bc00a710 comp rx=0 tx=0).stop 2024-03-19T01:24:20.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 shutdown_connections 2024-03-19T01:24:20.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f63a4067660 0x7f63a4069b20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63c013dff0 0x7f63c01de2a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c00b9bb0 0x7f63c0143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 --2- 172.21.15.7:0/1542806258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c00af4f0 0x7f63c0144a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:20.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 >> 172.21.15.7:0/1542806258 conn(0x7f63c006e1b0 msgr2=0x7f63c00b6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:20.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 shutdown_connections 2024-03-19T01:24:20.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:20.894+0000 7f63c7774640 1 -- 172.21.15.7:0/1542806258 wait complete. 2024-03-19T01:24:20.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:24:21.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f45076ed640 1 Processor -- start 2024-03-19T01:24:21.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f45076ed640 1 -- start start 2024-03-19T01:24:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f45076ed640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f450006d590 0x7f450006d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f45076ed640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450014b070 0x7f4500151c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f45076ed640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450006ce10 con 0x7f45000b5c10 2024-03-19T01:24:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f45076ed640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f4500147730 con 0x7f450006d590 2024-03-19T01:24:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f45076ed640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f45000bba90 con 0x7f450014b070 2024-03-19T01:24:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f4505489640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f450006d590 0x7f450006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f4505489640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f450006d590 0x7f450006d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34064/0 (socket says 172.21.15.7:34064) 2024-03-19T01:24:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f4505489640 1 -- 172.21.15.7:0/2790859619 learned_addr learned my addr 172.21.15.7:0/2790859619 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f4504c88640 1 --1- 172.21.15.7:0/2790859619 >> v1:172.21.15.7:6789/0 conn(0x7f45000b5c10 0x7f4500147bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37692/0 (socket says 172.21.15.7:37692) 2024-03-19T01:24:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f4505c8a640 1 --2- 172.21.15.7:0/2790859619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450014b070 0x7f4500151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f4505489640 1 -- 172.21.15.7:0/2790859619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450014b070 msgr2=0x7f4500151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f4505489640 1 --2- 172.21.15.7:0/2790859619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450014b070 0x7f4500151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f4505489640 1 -- 172.21.15.7:0/2790859619 >> v1:172.21.15.7:6789/0 conn(0x7f45000b5c10 legacy=0x7f4500147bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f4505489640 1 -- 172.21.15.7:0/2790859619 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4500147350 con 0x7f450006d590 2024-03-19T01:24:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.034+0000 7f44fffff640 1 -- 172.21.15.7:0/2790859619 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2263111020 0 0) 0x7f450006ce10 con 0x7f45000b5c10 2024-03-19T01:24:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f4505489640 1 --2- 172.21.15.7:0/2790859619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450006d590 0x7f450006d970 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009e50 tx=0x7f44f001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b23c381856cda4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f44fffff640 1 -- 172.21.15.7:0/2790859619 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f00a5660 con 0x7f450006d590 2024-03-19T01:24:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f44fffff640 1 -- 172.21.15.7:0/2790859619 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f00a5800 con 0x7f450006d590 2024-03-19T01:24:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f44fffff640 1 -- 172.21.15.7:0/2790859619 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f00a5bd0 con 0x7f450006d590 2024-03-19T01:24:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 -- 172.21.15.7:0/2790859619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450006d590 msgr2=0x7f450006d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 --2- 172.21.15.7:0/2790859619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450006d590 0x7f450006d970 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009e50 tx=0x7f44f001c9a0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 -- 172.21.15.7:0/2790859619 shutdown_connections 2024-03-19T01:24:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 --2- 172.21.15.7:0/2790859619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450014b070 0x7f4500151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 --2- 172.21.15.7:0/2790859619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450006d590 0x7f450006d970 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 -- 172.21.15.7:0/2790859619 >> 172.21.15.7:0/2790859619 conn(0x7f45000bace0 msgr2=0x7f45000bb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 -- 172.21.15.7:0/2790859619 shutdown_connections 2024-03-19T01:24:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 -- 172.21.15.7:0/2790859619 wait complete. 2024-03-19T01:24:21.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 Processor -- start 2024-03-19T01:24:21.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 -- start start 2024-03-19T01:24:21.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450006d590 0x7f45000b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45000b5c10 0x7f45001444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450014b070 0x7f450013dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4500154d20 con 0x7f450014b070 2024-03-19T01:24:21.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4500154ba0 con 0x7f45000b5c10 2024-03-19T01:24:21.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f45076ed640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4500154ea0 con 0x7f450006d590 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f4505c8a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450014b070 0x7f450013dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.038+0000 7f4505c8a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450014b070 0x7f450013dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34078/0 (socket says 172.21.15.7:34078) 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505c8a640 1 -- 172.21.15.7:0/79732137 learned_addr learned my addr 172.21.15.7:0/79732137 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4504c88640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45000b5c10 0x7f45001444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505489640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450006d590 0x7f45000b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505c8a640 1 -- 172.21.15.7:0/79732137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450006d590 msgr2=0x7f45000b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505c8a640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450006d590 0x7f45000b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505c8a640 1 -- 172.21.15.7:0/79732137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45000b5c10 msgr2=0x7f45001444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505c8a640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45000b5c10 0x7f45001444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505c8a640 1 -- 172.21.15.7:0/79732137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4500143d20 con 0x7f450014b070 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505c8a640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450014b070 0x7f450013dca0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f44f800d560 tx=0x7f44f800da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f44fdffb640 1 -- 172.21.15.7:0/79732137 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80125f0 con 0x7f450014b070 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f44fdffb640 1 -- 172.21.15.7:0/79732137 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f8019070 con 0x7f450014b070 2024-03-19T01:24:21.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45000aefa0 con 0x7f450014b070 2024-03-19T01:24:21.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f44fdffb640 1 -- 172.21.15.7:0/79732137 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80129f0 con 0x7f450014b070 2024-03-19T01:24:21.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45001430c0 con 0x7f450014b070 2024-03-19T01:24:21.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4500068630 con 0x7f450014b070 2024-03-19T01:24:21.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f44fdffb640 1 -- 172.21.15.7:0/79732137 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f44f8012b90 con 0x7f450014b070 2024-03-19T01:24:21.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f44fdffb640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44ec067690 0x7f44ec069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f44fdffb640 1 -- 172.21.15.7:0/79732137 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f44f8085c30 con 0x7f450014b070 2024-03-19T01:24:21.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505489640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44ec067690 0x7f44ec069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.042+0000 7f4505489640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44ec067690 0x7f44ec069b50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f44f00ae000 tx=0x7f44f00a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:21.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.046+0000 7f44fdffb640 1 -- 172.21.15.7:0/79732137 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f45000aefa0 con 0x7f450014b070 2024-03-19T01:24:21.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4500067c00 con 0x7f450014b070 2024-03-19T01:24:21.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f44fdffb640 1 -- 172.21.15.7:0/79732137 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) ==== 100+0+11 (secure 0 0 0) 0x7f44f80568a0 con 0x7f450014b070 2024-03-19T01:24:21.356 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:24:21.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44ec067690 msgr2=0x7f44ec069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44ec067690 0x7f44ec069b50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f44f00ae000 tx=0x7f44f00a7070 comp rx=0 tx=0).stop 2024-03-19T01:24:21.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450014b070 msgr2=0x7f450013dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450014b070 0x7f450013dca0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f44f800d560 tx=0x7f44f800da20 comp rx=0 tx=0).stop 2024-03-19T01:24:21.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 shutdown_connections 2024-03-19T01:24:21.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f44ec067690 0x7f44ec069b50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f450014b070 0x7f450013dca0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45000b5c10 0x7f45001444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 --2- 172.21.15.7:0/79732137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f450006d590 0x7f45000b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 >> 172.21.15.7:0/79732137 conn(0x7f45000bace0 msgr2=0x7f45001458d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:21.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 shutdown_connections 2024-03-19T01:24:21.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.354+0000 7f45076ed640 1 -- 172.21.15.7:0/79732137 wait complete. 2024-03-19T01:24:21.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T01:24:21.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.482+0000 7fab50a27640 1 Processor -- start 2024-03-19T01:24:21.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.482+0000 7fab50a27640 1 -- start start 2024-03-19T01:24:21.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.482+0000 7fab50a27640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fab4c144550 0x7fab4c0b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.482+0000 7fab50a27640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c0b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab50a27640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4c0b24b0 con 0x7fab4c0b20d0 2024-03-19T01:24:21.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab50a27640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fab4c0b4030 con 0x7fab4c144550 2024-03-19T01:24:21.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab50a27640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fab4c0b3400 con 0x7fab4c13dc50 2024-03-19T01:24:21.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab4ad9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c0b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab49d9b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fab4c144550 0x7fab4c0b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab4ad9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c0b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:39726/0 (socket says 172.21.15.7:39726) 2024-03-19T01:24:21.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab4ad9d640 1 -- 172.21.15.7:0/4047699157 learned_addr learned my addr 172.21.15.7:0/4047699157 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:21.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab49d9b640 1 -- 172.21.15.7:0/4047699157 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 msgr2=0x7fab4c0b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:21.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab49d9b640 1 --2- 172.21.15.7:0/4047699157 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c0b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab49d9b640 1 -- 172.21.15.7:0/4047699157 >> v1:172.21.15.167:6789/0 conn(0x7fab4c0b20d0 legacy=0x7fab4c13d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab49d9b640 1 -- 172.21.15.7:0/4047699157 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4c143a70 con 0x7fab4c144550 2024-03-19T01:24:21.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab4ad9d640 1 --2- 172.21.15.7:0/4047699157 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c0b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:21.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab49d9b640 1 --2- 172.21.15.7:0/4047699157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c144550 0x7fab4c0b1750 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab4000a990 tx=0x7fab4001d7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3c1d73a872ee22b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:21.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab4959a640 1 -- 172.21.15.7:0/4047699157 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab400ad070 con 0x7fab4c144550 2024-03-19T01:24:21.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab4959a640 1 -- 172.21.15.7:0/4047699157 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab400a7640 con 0x7fab4c144550 2024-03-19T01:24:21.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.486+0000 7fab4959a640 1 -- 172.21.15.7:0/4047699157 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4000b040 con 0x7fab4c144550 2024-03-19T01:24:21.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- 172.21.15.7:0/4047699157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c144550 msgr2=0x7fab4c0b1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 --2- 172.21.15.7:0/4047699157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c144550 0x7fab4c0b1750 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab4000a990 tx=0x7fab4001d7e0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- 172.21.15.7:0/4047699157 shutdown_connections 2024-03-19T01:24:21.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 --2- 172.21.15.7:0/4047699157 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c0b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 --2- 172.21.15.7:0/4047699157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c144550 0x7fab4c0b1750 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- 172.21.15.7:0/4047699157 >> 172.21.15.7:0/4047699157 conn(0x7fab4c0b5bf0 msgr2=0x7fab4c0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:21.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- 172.21.15.7:0/4047699157 shutdown_connections 2024-03-19T01:24:21.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- 172.21.15.7:0/4047699157 wait complete. 2024-03-19T01:24:21.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 Processor -- start 2024-03-19T01:24:21.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- start start 2024-03-19T01:24:21.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c0b20d0 0x7fab4c1365f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c138980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4c144550 0x7fab4c0af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c150370 con 0x7fab4c144550 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c1501f0 con 0x7fab4c0b20d0 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fab4c1504f0 con 0x7fab4c13dc50 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4c144550 0x7fab4c0af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4c144550 0x7fab4c0af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34082/0 (socket says 172.21.15.7:34082) 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4ad9d640 1 -- 172.21.15.7:0/3020937010 learned_addr learned my addr 172.21.15.7:0/3020937010 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab49d9b640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c138980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4ad9d640 1 -- 172.21.15.7:0/3020937010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 msgr2=0x7fab4c138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4a59c640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c0b20d0 0x7fab4c1365f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4ad9d640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c138980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4ad9d640 1 -- 172.21.15.7:0/3020937010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c0b20d0 msgr2=0x7fab4c1365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4ad9d640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c0b20d0 0x7fab4c1365f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4ad9d640 1 -- 172.21.15.7:0/3020937010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4c1e4110 con 0x7fab4c144550 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4a59c640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c0b20d0 0x7fab4c1365f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab4ad9d640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4c144550 0x7fab4c0af890 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fab4400efc0 tx=0x7fab4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab3affd640 1 -- 172.21.15.7:0/3020937010 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4400cb80 con 0x7fab4c144550 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab3affd640 1 -- 172.21.15.7:0/3020937010 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab4400cd20 con 0x7fab4c144550 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab4c1e3630 con 0x7fab4c144550 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab3affd640 1 -- 172.21.15.7:0/3020937010 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44003bc0 con 0x7fab4c144550 2024-03-19T01:24:21.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.490+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab4c1e3bf0 con 0x7fab4c144550 2024-03-19T01:24:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.494+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab4c075460 con 0x7fab4c144550 2024-03-19T01:24:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.494+0000 7fab3affd640 1 -- 172.21.15.7:0/3020937010 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fab4401b030 con 0x7fab4c144550 2024-03-19T01:24:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.494+0000 7fab3affd640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:21.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.494+0000 7fab3affd640 1 -- 172.21.15.7:0/3020937010 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fab44085c90 con 0x7fab4c144550 2024-03-19T01:24:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.494+0000 7fab4a59c640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:21.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.494+0000 7fab4a59c640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab3c000f70 tx=0x7fab3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:21.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.498+0000 7fab3affd640 1 -- 172.21.15.7:0/3020937010 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fab44022020 con 0x7fab4c144550 2024-03-19T01:24:21.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.710+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fab4c141280 con 0x7fab4c144550 2024-03-19T01:24:21.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.710+0000 7fab3affd640 1 -- 172.21.15.7:0/3020937010 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v30) ==== 80+0+0 (secure 0 0 0) 0x7fab44010040 con 0x7fab4c144550 2024-03-19T01:24:21.713 INFO:teuthology.orchestra.run.smithi007.stderr:listed 0 entries 2024-03-19T01:24:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 msgr2=0x7fab2c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab3c000f70 tx=0x7fab3c009040 comp rx=0 tx=0).stop 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4c144550 msgr2=0x7fab4c0af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4c144550 0x7fab4c0af890 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fab4400efc0 tx=0x7fab4400c490 comp rx=0 tx=0).stop 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 shutdown_connections 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab4c13dc50 0x7fab4c138980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4c144550 0x7fab4c0af890 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 --2- 172.21.15.7:0/3020937010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4c0b20d0 0x7fab4c1365f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 >> 172.21.15.7:0/3020937010 conn(0x7fab4c0b5bf0 msgr2=0x7fab4c13d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 shutdown_connections 2024-03-19T01:24:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:21.714+0000 7fab50a27640 1 -- 172.21.15.7:0/3020937010 wait complete. 2024-03-19T01:24:21.731 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi007.front.sepia.ceph.com... 2024-03-19T01:24:21.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:21.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-19T01:24:21.739 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi007.stderr:ceph-fuse[36292]: fuse finished with error 0 and tester_r 0 2024-03-19T01:24:21.740 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:24:21.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:21.741 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-19T01:24:21.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:24:22.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.062+0000 7ff377070640 1 Processor -- start 2024-03-19T01:24:22.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.062+0000 7ff377070640 1 -- start start 2024-03-19T01:24:22.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff377070640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff370074390 0x7ff370141430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff377070640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3700add50 0x7ff370147830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff377070640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3700aead0 con 0x7ff370141970 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff377070640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7ff370074770 con 0x7ff370074390 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff377070640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff3700af050 con 0x7ff3700add50 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff37560d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff370141970 0x7ff3700b3af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37704/0 (socket says 172.21.15.7:37704) 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff37560d640 1 -- 172.21.15.7:0/2010006857 learned_addr learned my addr 172.21.15.7:0/2010006857 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff374e0c640 1 --2- 172.21.15.7:0/2010006857 >> v2:172.21.15.167:3300/0 conn(0x7ff370074390 0x7ff370141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36ffff640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3700add50 0x7ff370147830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2880614225 0 0) 0x7ff3700aead0 con 0x7ff370141970 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff358003530 con 0x7ff370141970 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3980874178 0 0) 0x7ff358003530 con 0x7ff370141970 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3700add50 msgr2=0x7ff370147830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3700add50 0x7ff370147830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 >> v2:172.21.15.167:3300/0 conn(0x7ff370074390 msgr2=0x7ff370141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 --2- 172.21.15.7:0/2010006857 >> v2:172.21.15.167:3300/0 conn(0x7ff370074390 0x7ff370141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3700aed70 con 0x7ff370141970 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7ff3680016a0 con 0x7ff370141970 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 >> v1:172.21.15.7:6789/0 conn(0x7ff370141970 legacy=0x7ff3700b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3580039c0 0x7ff358003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3700add50 0x7ff358004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3580051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff370074770 con 0x7ff358004d50 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff3700af050 con 0x7ff3580039c0 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff35800ba50 con 0x7ff3700add50 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff37560d640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3580051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff374e0c640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3580039c0 0x7ff358003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff37560d640 1 -- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3700add50 msgr2=0x7ff358004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff37560d640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3700add50 0x7ff358004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff37560d640 1 -- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3580039c0 msgr2=0x7ff358003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff37560d640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3580039c0 0x7ff358003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff37560d640 1 -- 172.21.15.7:0/2010006857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff358003530 con 0x7ff358004d50 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.066+0000 7ff37560d640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3580051d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff3700aeca0 tx=0x7ff36801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4f3bef189550b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff368007d80 con 0x7ff358004d50 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3680a9070 con 0x7ff358004d50 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff36f7fe640 1 -- 172.21.15.7:0/2010006857 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff368005d40 con 0x7ff358004d50 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- 172.21.15.7:0/2010006857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 msgr2=0x7ff3580051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 --2- 172.21.15.7:0/2010006857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3580051d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff3700aeca0 tx=0x7ff36801cc60 comp rx=0 tx=0).stop 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff37560d640 1 -- 172.21.15.7:0/2010006857 reap_dead start 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- 172.21.15.7:0/2010006857 shutdown_connections 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- 172.21.15.7:0/2010006857 >> 172.21.15.7:0/2010006857 conn(0x7ff3700b5bf0 msgr2=0x7ff3700b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- 172.21.15.7:0/2010006857 shutdown_connections 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- 172.21.15.7:0/2010006857 wait complete. 2024-03-19T01:24:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 Processor -- start 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- start start 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3701e31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff370139100 0x7ff3700b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff37013d630 0x7ff370134410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff3701479b0 con 0x7ff358004d50 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff370147830 con 0x7ff370139100 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff370147b30 con 0x7ff37013d630 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff374e0c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3701e31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff374e0c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3701e31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34102/0 (socket says 172.21.15.7:34102) 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff374e0c640 1 -- 172.21.15.7:0/2989079286 learned_addr learned my addr 172.21.15.7:0/2989079286 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff36ffff640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff370139100 0x7ff3700b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff37560d640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff37013d630 0x7ff370134410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff374e0c640 1 -- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff37013d630 msgr2=0x7ff370134410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff374e0c640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff37013d630 0x7ff370134410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff374e0c640 1 -- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff370139100 msgr2=0x7ff3700b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff374e0c640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff370139100 0x7ff3700b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff374e0c640 1 -- 172.21.15.7:0/2989079286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37013afe0 con 0x7ff358004d50 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff37560d640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff37013d630 0x7ff370134410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff36ffff640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff370139100 0x7ff3700b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff374e0c640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3701e31a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff360007f30 tx=0x7ff360011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff36d7fa640 1 -- 172.21.15.7:0/2989079286 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36001a020 con 0x7ff358004d50 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff36d7fa640 1 -- 172.21.15.7:0/2989079286 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff360016470 con 0x7ff358004d50 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff370139650 con 0x7ff358004d50 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff36d7fa640 1 -- 172.21.15.7:0/2989079286 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff360016860 con 0x7ff358004d50 2024-03-19T01:24:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.070+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3700b33d0 con 0x7ff358004d50 2024-03-19T01:24:22.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.074+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff37013b4e0 con 0x7ff358004d50 2024-03-19T01:24:22.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.074+0000 7ff36d7fa640 1 -- 172.21.15.7:0/2989079286 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff36001d020 con 0x7ff358004d50 2024-03-19T01:24:22.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.074+0000 7ff36d7fa640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff35c067660 0x7ff35c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.074+0000 7ff36d7fa640 1 -- 172.21.15.7:0/2989079286 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7ff3600867e0 con 0x7ff358004d50 2024-03-19T01:24:22.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.074+0000 7ff36ffff640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff35c067660 0x7ff35c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.074+0000 7ff36ffff640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff35c067660 0x7ff35c069b20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff3640047c0 tx=0x7ff364008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:22.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.078+0000 7ff36d7fa640 1 -- 172.21.15.7:0/2989079286 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff37013b4e0 con 0x7ff358004d50 2024-03-19T01:24:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.266+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff37014db20 con 0x7ff358004d50 2024-03-19T01:24:22.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.266+0000 7ff36d7fa640 1 -- 172.21.15.7:0/2989079286 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3531 (secure 0 0 0) 0x7ff360024030 con 0x7ff358004d50 2024-03-19T01:24:22.271 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:22.271 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4753,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/3744754940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3744754940},{"type":"v1","addr":"172.21.15.167:6833","nonce":3744754940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:24:10.251824+0000","modified":"2024-03-19T01:24:13.280725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5001":{"gid":5001,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/4260183338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4260183338},{"type":"v1","addr":"172.21.15.7:6835","nonce":4260183338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5001,"qdb_cluster":[5001]},"id":2}]} 2024-03-19T01:24:22.271 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 16 2024-03-19T01:24:22.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff35c067660 msgr2=0x7ff35c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff35c067660 0x7ff35c069b20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff3640047c0 tx=0x7ff364008040 comp rx=0 tx=0).stop 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 msgr2=0x7ff3701e31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3701e31a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff360007f30 tx=0x7ff360011660 comp rx=0 tx=0).stop 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 shutdown_connections 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff35c067660 0x7ff35c069b20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff37013d630 0x7ff370134410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff370139100 0x7ff3700b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 --2- 172.21.15.7:0/2989079286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff358004d50 0x7ff3701e31a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 >> 172.21.15.7:0/2989079286 conn(0x7ff3700b5bf0 msgr2=0x7ff3700b8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 shutdown_connections 2024-03-19T01:24:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.270+0000 7ff377070640 1 -- 172.21.15.7:0/2989079286 wait complete. 2024-03-19T01:24:22.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:22.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fceda35d640 1 Processor -- start 2024-03-19T01:24:22.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fceda35d640 1 -- start start 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fceda35d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced406b520 0x7fced414acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fceda35d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fced414b230 0x7fced413d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fceda35d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fced406b900 con 0x7fced413de00 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fceda35d640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fced40716a0 con 0x7fced414b230 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fceda35d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fced4072800 con 0x7fced406b520 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced88fa640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fced413de00 0x7fced40b9a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37706/0 (socket says 172.21.15.7:37706) 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced88fa640 1 -- 172.21.15.7:0/1433199495 learned_addr learned my addr 172.21.15.7:0/1433199495 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced37fe640 1 --2- 172.21.15.7:0/1433199495 >> v2:172.21.15.167:3300/0 conn(0x7fced414b230 0x7fced413d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced3fff640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced406b520 0x7fced414acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3027512296 0 0) 0x7fced406b900 con 0x7fced413de00 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcebc003530 con 0x7fced413de00 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3877335103 0 0) 0x7fcebc003530 con 0x7fced413de00 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced406b520 msgr2=0x7fced414acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced406b520 0x7fced414acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 >> v2:172.21.15.167:3300/0 conn(0x7fced414b230 msgr2=0x7fced413d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 --2- 172.21.15.7:0/1433199495 >> v2:172.21.15.167:3300/0 conn(0x7fced414b230 0x7fced413d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced414b670 con 0x7fced413de00 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fcecc001a00 con 0x7fced413de00 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 >> v1:172.21.15.7:6789/0 conn(0x7fced413de00 legacy=0x7fced40b9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcebc0039c0 0x7fcebc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced406b520 0x7fcebc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc004d50 0x7fcebc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc00b9d0 con 0x7fcebc0039c0 2024-03-19T01:24:22.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc00bb50 con 0x7fcebc004d50 2024-03-19T01:24:22.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcebc00bd00 con 0x7fced406b520 2024-03-19T01:24:22.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced3fff640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcebc0039c0 0x7fcebc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced37fe640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced406b520 0x7fcebc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced88fa640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc004d50 0x7fcebc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced3fff640 1 -- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced406b520 msgr2=0x7fcebc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced3fff640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced406b520 0x7fcebc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced3fff640 1 -- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc004d50 msgr2=0x7fcebc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced3fff640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc004d50 0x7fcebc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.398+0000 7fced3fff640 1 -- 172.21.15.7:0/1433199495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc003530 con 0x7fcebc0039c0 2024-03-19T01:24:22.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced3fff640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcebc0039c0 0x7fcebc003de0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009ae0 tx=0x7fcec401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc14fc839b51c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:22.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec40b2070 con 0x7fcebc0039c0 2024-03-19T01:24:22.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcec40a5550 con 0x7fcebc0039c0 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced2ffd640 1 -- 172.21.15.7:0/1433199495 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec40096b0 con 0x7fcebc0039c0 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- 172.21.15.7:0/1433199495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcebc0039c0 msgr2=0x7fcebc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 --2- 172.21.15.7:0/1433199495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcebc0039c0 0x7fcebc003de0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009ae0 tx=0x7fcec401ca70 comp rx=0 tx=0).stop 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced88fa640 1 -- 172.21.15.7:0/1433199495 reap_dead start 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- 172.21.15.7:0/1433199495 shutdown_connections 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- 172.21.15.7:0/1433199495 >> 172.21.15.7:0/1433199495 conn(0x7fced4074130 msgr2=0x7fced4074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- 172.21.15.7:0/1433199495 shutdown_connections 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- 172.21.15.7:0/1433199495 wait complete. 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 Processor -- start 2024-03-19T01:24:22.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- start start 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc0039c0 0x7fced4141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced413efc0 0x7fced41e3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced41e3b80 0x7fced413a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fced4141a30 con 0x7fced413efc0 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fced40718d0 con 0x7fcebc0039c0 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fced4070700 con 0x7fced41e3b80 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced413efc0 0x7fced41e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced413efc0 0x7fced41e3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34124/0 (socket says 172.21.15.7:34124) 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced37fe640 1 -- 172.21.15.7:0/891340177 learned_addr learned my addr 172.21.15.7:0/891340177 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced3fff640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc0039c0 0x7fced4141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced88fa640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced41e3b80 0x7fced413a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced37fe640 1 -- 172.21.15.7:0/891340177 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced41e3b80 msgr2=0x7fced413a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced37fe640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced41e3b80 0x7fced413a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced37fe640 1 -- 172.21.15.7:0/891340177 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc0039c0 msgr2=0x7fced4141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced37fe640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc0039c0 0x7fced4141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced37fe640 1 -- 172.21.15.7:0/891340177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced41403d0 con 0x7fced413efc0 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced3fff640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc0039c0 0x7fced4141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced37fe640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced413efc0 0x7fced41e3640 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcec8011690 tx=0x7fcec8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced0ff9640 1 -- 172.21.15.7:0/891340177 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec800f9c0 con 0x7fced413efc0 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fced0ff9640 1 -- 172.21.15.7:0/891340177 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcec800fb60 con 0x7fced413efc0 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.402+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced41d2a90 con 0x7fced413efc0 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.406+0000 7fced0ff9640 1 -- 172.21.15.7:0/891340177 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec8015460 con 0x7fced413efc0 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.406+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced413fd00 con 0x7fced413efc0 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.406+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fced406c350 con 0x7fced413efc0 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.406+0000 7fced0ff9640 1 -- 172.21.15.7:0/891340177 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcec8015600 con 0x7fced413efc0 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.406+0000 7fced0ff9640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcec0067660 0x7fcec0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.406+0000 7fced0ff9640 1 -- 172.21.15.7:0/891340177 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fcec8086190 con 0x7fced413efc0 2024-03-19T01:24:22.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.406+0000 7fced3fff640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcec0067660 0x7fcec0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.406+0000 7fced3fff640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcec0067660 0x7fcec0069b20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcec40af000 tx=0x7fcec40046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.410+0000 7fced0ff9640 1 -- 172.21.15.7:0/891340177 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fced41d2a90 con 0x7fced413efc0 2024-03-19T01:24:22.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.570+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fced4144e50 con 0x7fced413efc0 2024-03-19T01:24:22.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.570+0000 7fced0ff9640 1 -- 172.21.15.7:0/891340177 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16633 (secure 0 0 0) 0x7fced4144e50 con 0x7fced413efc0 2024-03-19T01:24:22.575 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:22.575 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":30,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:24:11.876754+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:24:08.795930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-19T01:24:09.233323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcec0067660 msgr2=0x7fcec0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcec0067660 0x7fcec0069b20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcec40af000 tx=0x7fcec40046d0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced413efc0 msgr2=0x7fced41e3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced413efc0 0x7fced41e3640 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcec8011690 tx=0x7fcec8011b50 comp rx=0 tx=0).stop 2024-03-19T01:24:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 shutdown_connections 2024-03-19T01:24:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcec0067660 0x7fcec0069b20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fced41e3b80 0x7fced413a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced413efc0 0x7fced41e3640 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 --2- 172.21.15.7:0/891340177 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcebc0039c0 0x7fced4141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 >> 172.21.15.7:0/891340177 conn(0x7fced4074130 msgr2=0x7fced40b6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 shutdown_connections 2024-03-19T01:24:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.574+0000 7fceda35d640 1 -- 172.21.15.7:0/891340177 wait complete. 2024-03-19T01:24:22.594 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:23:14.969421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:24:08.795930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:24:09.233323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:24:22.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:24:22.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01fce04640 1 Processor -- start 2024-03-19T01:24:22.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01fce04640 1 -- start start 2024-03-19T01:24:22.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01fce04640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01f8074390 0x7f01f80b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01fce04640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01f813df80 0x7f01f813e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01fce04640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f01f81440c0 0x7f01f8144530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01fce04640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f01f80b2140 con 0x7f01f8074390 2024-03-19T01:24:22.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01fce04640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f01f80b56e0 con 0x7f01f81440c0 2024-03-19T01:24:22.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01fce04640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f01f806fef0 con 0x7f01f813df80 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01f8074390 0x7f01f80b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01f8074390 0x7f01f80b1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39254/0 (socket says 172.21.15.7:39254) 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f659c640 1 -- 172.21.15.7:0/2285344372 learned_addr learned my addr 172.21.15.7:0/2285344372 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f6d9d640 1 --2- 172.21.15.7:0/2285344372 >> v2:172.21.15.167:3300/0 conn(0x7f01f81440c0 0x7f01f8144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f5d9b640 1 --2- 172.21.15.7:0/2285344372 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01f813df80 0x7f01f813e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f659c640 1 -- 172.21.15.7:0/2285344372 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01f813df80 msgr2=0x7f01f813e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f659c640 1 --2- 172.21.15.7:0/2285344372 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01f813df80 0x7f01f813e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f659c640 1 -- 172.21.15.7:0/2285344372 >> v2:172.21.15.167:3300/0 conn(0x7f01f81440c0 msgr2=0x7f01f8144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f659c640 1 --2- 172.21.15.7:0/2285344372 >> v2:172.21.15.167:3300/0 conn(0x7f01f81440c0 0x7f01f8144530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.710+0000 7f01f659c640 1 -- 172.21.15.7:0/2285344372 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f80b2360 con 0x7f01f8074390 2024-03-19T01:24:22.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 --2- 172.21.15.7:0/2285344372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 0x7f01f80b1c00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f01f80b6ca0 tx=0x7f01e801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ecdcf6d98fd94 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:22.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f559a640 1 -- 172.21.15.7:0/2285344372 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e80ab650 con 0x7f01f8074390 2024-03-19T01:24:22.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f559a640 1 -- 172.21.15.7:0/2285344372 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e80b0070 con 0x7f01f8074390 2024-03-19T01:24:22.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f559a640 1 -- 172.21.15.7:0/2285344372 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e80aba10 con 0x7f01f8074390 2024-03-19T01:24:22.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- 172.21.15.7:0/2285344372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 msgr2=0x7f01f80b1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 --2- 172.21.15.7:0/2285344372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 0x7f01f80b1c00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f01f80b6ca0 tx=0x7f01e801c490 comp rx=0 tx=0).stop 2024-03-19T01:24:22.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- 172.21.15.7:0/2285344372 shutdown_connections 2024-03-19T01:24:22.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 --2- 172.21.15.7:0/2285344372 >> v2:172.21.15.167:3300/0 conn(0x7f01f81440c0 0x7f01f8144530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 --2- 172.21.15.7:0/2285344372 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01f813df80 0x7f01f813e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 --2- 172.21.15.7:0/2285344372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 0x7f01f80b1c00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- 172.21.15.7:0/2285344372 >> 172.21.15.7:0/2285344372 conn(0x7f01f80b5c90 msgr2=0x7f01f80b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:22.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- 172.21.15.7:0/2285344372 shutdown_connections 2024-03-19T01:24:22.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- 172.21.15.7:0/2285344372 wait complete. 2024-03-19T01:24:22.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 Processor -- start 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- start start 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 0x7f01f81461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01f813df80 0x7f01f81e3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01f81440c0 0x7f01f8071630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f01f80b2b70 con 0x7f01f8074390 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f01f80b29f0 con 0x7f01f813df80 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f01f8147e40 con 0x7f01f81440c0 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 0x7f01f81461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 0x7f01f81461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39268/0 (socket says 172.21.15.7:39268) 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 -- 172.21.15.7:0/878933606 learned_addr learned my addr 172.21.15.7:0/878933606 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 -- 172.21.15.7:0/878933606 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01f81440c0 msgr2=0x7f01f8071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01f81440c0 0x7f01f8071630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 -- 172.21.15.7:0/878933606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01f813df80 msgr2=0x7f01f81e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01f813df80 0x7f01f81e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 -- 172.21.15.7:0/878933606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f81e2870 con 0x7f01f8074390 2024-03-19T01:24:22.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01f659c640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 0x7f01f81461e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f01e80ad040 tx=0x7f01e801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:22.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01e6ffd640 1 -- 172.21.15.7:0/878933606 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e80b4800 con 0x7f01f8074390 2024-03-19T01:24:22.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01e6ffd640 1 -- 172.21.15.7:0/878933606 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e80b49c0 con 0x7f01f8074390 2024-03-19T01:24:22.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01e6ffd640 1 -- 172.21.15.7:0/878933606 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e80b4db0 con 0x7f01f8074390 2024-03-19T01:24:22.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f80b8b60 con 0x7f01f8074390 2024-03-19T01:24:22.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.714+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f81e35b0 con 0x7f01f8074390 2024-03-19T01:24:22.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.718+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01f8075570 con 0x7f01f8074390 2024-03-19T01:24:22.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.718+0000 7f01e6ffd640 1 -- 172.21.15.7:0/878933606 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f01e80c7440 con 0x7f01f8074390 2024-03-19T01:24:22.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.718+0000 7f01e6ffd640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01d8067690 0x7f01d8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:22.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.718+0000 7f01e6ffd640 1 -- 172.21.15.7:0/878933606 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f01e81206b0 con 0x7f01f8074390 2024-03-19T01:24:22.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.718+0000 7f01f5d9b640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01d8067690 0x7f01d8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:22.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.718+0000 7f01f5d9b640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01d8067690 0x7f01d8069b50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f01f80b62e0 tx=0x7f01ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:22.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.722+0000 7f01e6ffd640 1 -- 172.21.15.7:0/878933606 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f01e8016110 con 0x7f01f8074390 2024-03-19T01:24:22.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.878+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01f806c2d0 con 0x7f01f8074390 2024-03-19T01:24:22.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.878+0000 7f01e6ffd640 1 -- 172.21.15.7:0/878933606 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7f01e80f14a0 con 0x7f01f8074390 2024-03-19T01:24:22.882 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01d8067690 msgr2=0x7f01d8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01d8067690 0x7f01d8069b50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f01f80b62e0 tx=0x7f01ec008040 comp rx=0 tx=0).stop 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 msgr2=0x7f01f81461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 0x7f01f81461e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f01e80ad040 tx=0x7f01e801acd0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 shutdown_connections 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f01d8067690 0x7f01d8069b50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01f81440c0 0x7f01f8071630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01f813df80 0x7f01f81e3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 --2- 172.21.15.7:0/878933606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01f8074390 0x7f01f81461e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:22.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 >> 172.21.15.7:0/878933606 conn(0x7f01f80b5c90 msgr2=0x7f01f813c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:22.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 shutdown_connections 2024-03-19T01:24:22.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:22.882+0000 7f01fce04640 1 -- 172.21.15.7:0/878933606 wait complete. 2024-03-19T01:24:22.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:24:23.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.010+0000 7f4d38ac4640 1 Processor -- start 2024-03-19T01:24:23.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.010+0000 7f4d38ac4640 1 -- start start 2024-03-19T01:24:23.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.010+0000 7f4d38ac4640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f4d340b6960 0x7f4d340b6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d38ac4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d340b56e0 con 0x7f4d34145f90 2024-03-19T01:24:23.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d38ac4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d340b59c0 con 0x7f4d34145b30 2024-03-19T01:24:23.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d38ac4640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f4d341456a0 con 0x7f4d340b6960 2024-03-19T01:24:23.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d32d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4d34145f90 0x7f4d3414fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35422/0 (socket says 172.21.15.7:35422) 2024-03-19T01:24:23.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d32d9d640 1 -- 172.21.15.7:0/2217003918 learned_addr learned my addr 172.21.15.7:0/2217003918 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:23.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d3259c640 1 --2- 172.21.15.7:0/2217003918 >> v2:172.21.15.167:3300/0 conn(0x7f4d340b6960 0x7f4d340b6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545626304 0 0) 0x7f4d340b56e0 con 0x7f4d34145f90 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d20003530 con 0x7f4d34145f90 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3242401252 0 0) 0x7f4d20003530 con 0x7f4d34145f90 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 >> v2:172.21.15.167:3300/0 conn(0x7f4d340b6960 msgr2=0x7f4d340b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 --2- 172.21.15.7:0/2217003918 >> v2:172.21.15.167:3300/0 conn(0x7f4d340b6960 0x7f4d340b6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 >> v1:172.21.15.167:6789/0 conn(0x7f4d34145b30 legacy=0x7f4d340b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d341452c0 con 0x7f4d34145f90 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f4d2c001a00 con 0x7f4d34145f90 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 >> v1:172.21.15.7:6789/0 conn(0x7f4d34145f90 legacy=0x7f4d3414fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d200039c0 0x7f4d20003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d20004810 0x7f4d20004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d200051d0 0x7f4d2000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4d341456a0 con 0x7f4d20004810 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2000c170 con 0x7f4d200039c0 2024-03-19T01:24:23.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2000c370 con 0x7f4d200051d0 2024-03-19T01:24:23.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d29d9b640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d20004810 0x7f4d20004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d3259c640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d200039c0 0x7f4d20003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d32d9d640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d200051d0 0x7f4d2000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d29d9b640 1 -- 172.21.15.7:0/2217003918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d200051d0 msgr2=0x7f4d2000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:23.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d29d9b640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d200051d0 0x7f4d2000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d29d9b640 1 -- 172.21.15.7:0/2217003918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d200039c0 msgr2=0x7f4d20003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:23.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d29d9b640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d200039c0 0x7f4d20003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d29d9b640 1 -- 172.21.15.7:0/2217003918 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d20003530 con 0x7f4d20004810 2024-03-19T01:24:23.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d32d9d640 1 -- 172.21.15.7:0/2217003918 reap_dead start 2024-03-19T01:24:23.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d29d9b640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d20004810 0x7f4d20004c90 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4d24001780 tx=0x7f4d2401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d042f4a0769ef66 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:23.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d240013a0 con 0x7f4d20004810 2024-03-19T01:24:23.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d240052f0 con 0x7f4d20004810 2024-03-19T01:24:23.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.014+0000 7f4d31d9b640 1 -- 172.21.15.7:0/2217003918 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24003840 con 0x7f4d20004810 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- 172.21.15.7:0/2217003918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d20004810 msgr2=0x7f4d20004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d20004810 0x7f4d20004c90 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4d24001780 tx=0x7f4d2401c400 comp rx=0 tx=0).stop 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- 172.21.15.7:0/2217003918 shutdown_connections 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 --2- 172.21.15.7:0/2217003918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d20004810 0x7f4d20004c90 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- 172.21.15.7:0/2217003918 >> 172.21.15.7:0/2217003918 conn(0x7f4d340bace0 msgr2=0x7f4d340bb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- 172.21.15.7:0/2217003918 shutdown_connections 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- 172.21.15.7:0/2217003918 wait complete. 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 Processor -- start 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- start start 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d20004810 0x7f4d3413e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34141200 0x7f4d3413fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d34142910 0x7f4d3413cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4d34148830 con 0x7f4d34141200 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4d340b4ec0 con 0x7f4d34142910 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4d341489b0 con 0x7f4d20004810 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d29d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34141200 0x7f4d3413fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d3259c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d20004810 0x7f4d3413e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d29d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34141200 0x7f4d3413fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39284/0 (socket says 172.21.15.7:39284) 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d29d9b640 1 -- 172.21.15.7:0/551539052 learned_addr learned my addr 172.21.15.7:0/551539052 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d32d9d640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d34142910 0x7f4d3413cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d29d9b640 1 -- 172.21.15.7:0/551539052 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d20004810 msgr2=0x7f4d3413e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d29d9b640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d20004810 0x7f4d3413e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d29d9b640 1 -- 172.21.15.7:0/551539052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d34142910 msgr2=0x7f4d3413cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d29d9b640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d34142910 0x7f4d3413cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d29d9b640 1 -- 172.21.15.7:0/551539052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d34142480 con 0x7f4d34141200 2024-03-19T01:24:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d29d9b640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34141200 0x7f4d3413fc80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4d2401c230 tx=0x7f4d20005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d2b7fe640 1 -- 172.21.15.7:0/551539052 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d240b8020 con 0x7f4d34141200 2024-03-19T01:24:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d2b7fe640 1 -- 172.21.15.7:0/551539052 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d240054a0 con 0x7f4d34141200 2024-03-19T01:24:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d341d29f0 con 0x7f4d34141200 2024-03-19T01:24:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d2b7fe640 1 -- 172.21.15.7:0/551539052 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d240013a0 con 0x7f4d34141200 2024-03-19T01:24:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d341d1c10 con 0x7f4d34141200 2024-03-19T01:24:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.018+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d34140e50 con 0x7f4d34141200 2024-03-19T01:24:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.022+0000 7f4d2b7fe640 1 -- 172.21.15.7:0/551539052 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4d240bb020 con 0x7f4d34141200 2024-03-19T01:24:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.022+0000 7f4d2b7fe640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4d14067660 0x7f4d14069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.022+0000 7f4d3259c640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4d14067660 0x7f4d14069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.022+0000 7f4d2b7fe640 1 -- 172.21.15.7:0/551539052 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f4d2411f6f0 con 0x7f4d34141200 2024-03-19T01:24:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.022+0000 7f4d3259c640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4d14067660 0x7f4d14069b20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d3413ed80 tx=0x7f4d1c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.026+0000 7f4d2b7fe640 1 -- 172.21.15.7:0/551539052 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4d34140e50 con 0x7f4d34141200 2024-03-19T01:24:23.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:24:23.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:24:23.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4d34075570 con 0x7f4d34141200 2024-03-19T01:24:23.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d2b7fe640 1 -- 172.21.15.7:0/551539052 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) ==== 104+0+11 (secure 0 0 0) 0x7f4d240c2030 con 0x7f4d34141200 2024-03-19T01:24:23.190 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-19T01:24:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4d14067660 msgr2=0x7f4d14069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4d14067660 0x7f4d14069b20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d3413ed80 tx=0x7f4d1c009190 comp rx=0 tx=0).stop 2024-03-19T01:24:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34141200 msgr2=0x7f4d3413fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34141200 0x7f4d3413fc80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4d2401c230 tx=0x7f4d20005840 comp rx=0 tx=0).stop 2024-03-19T01:24:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 shutdown_connections 2024-03-19T01:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f4d14067660 0x7f4d14069b20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d34142910 0x7f4d3413cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34141200 0x7f4d3413fc80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 --2- 172.21.15.7:0/551539052 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d20004810 0x7f4d3413e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.186+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 >> 172.21.15.7:0/551539052 conn(0x7f4d340bace0 msgr2=0x7f4d340b5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.190+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 shutdown_connections 2024-03-19T01:24:23.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.190+0000 7f4d38ac4640 1 -- 172.21.15.7:0/551539052 wait complete. 2024-03-19T01:24:23.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:24:23.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.334+0000 7f05b2fee640 1 Processor -- start 2024-03-19T01:24:23.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.334+0000 7f05b2fee640 1 -- start start 2024-03-19T01:24:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b2fee640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac0b77f0 0x7f05ac0b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b2fee640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05ac0b2c20 con 0x7f05ac0b6e50 2024-03-19T01:24:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b2fee640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05ac0b5d00 con 0x7f05ac0b69f0 2024-03-19T01:24:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b2fee640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f05ac0b5fe0 con 0x7f05ac0b77f0 2024-03-19T01:24:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b158b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f05ac0b6e50 0x7f05ac14fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35438/0 (socket says 172.21.15.7:35438) 2024-03-19T01:24:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b158b640 1 -- 172.21.15.7:0/1721610603 learned_addr learned my addr 172.21.15.7:0/1721610603 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3980587081 0 0) 0x7f05ac0b2c20 con 0x7f05ac0b6e50 2024-03-19T01:24:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f059c003530 con 0x7f05ac0b6e50 2024-03-19T01:24:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b0d8a640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac0b77f0 0x7f05ac0b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3902503559 0 0) 0x7f059c003530 con 0x7f05ac0b6e50 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac0b77f0 msgr2=0x7f05ac0b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac0b77f0 0x7f05ac0b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 >> v1:172.21.15.167:6789/0 conn(0x7f05ac0b69f0 legacy=0x7f05ac0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ac0b64d0 con 0x7f05ac0b6e50 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f05a4001a00 con 0x7f05ac0b6e50 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 >> v1:172.21.15.7:6789/0 conn(0x7f05ac0b6e50 legacy=0x7f05ac14fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac0b77f0 0x7f059c0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f059c004020 0x7f059c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004d50 0x7f059c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac0b5fe0 con 0x7f059c004020 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f059c00bc10 con 0x7f059c004d50 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f059c00be10 con 0x7f05ac0b77f0 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05abfff640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f059c004020 0x7f059c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b0d8a640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac0b77f0 0x7f059c0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b158b640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004d50 0x7f059c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05abfff640 1 -- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac0b77f0 msgr2=0x7f059c0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05abfff640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac0b77f0 0x7f059c0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05abfff640 1 -- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004d50 msgr2=0x7f059c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05abfff640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004d50 0x7f059c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05abfff640 1 -- 172.21.15.7:0/1721610603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059c003530 con 0x7f059c004020 2024-03-19T01:24:23.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.338+0000 7f05b158b640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004d50 0x7f059c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05abfff640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f059c004020 0x7f059c004810 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0598001780 tx=0x7f059801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b1c8aef515e677 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05980013a0 con 0x7f059c004020 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05980052f0 con 0x7f059c004020 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05ab7fe640 1 -- 172.21.15.7:0/1721610603 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598003840 con 0x7f059c004020 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- 172.21.15.7:0/1721610603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f059c004020 msgr2=0x7f059c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 --2- 172.21.15.7:0/1721610603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f059c004020 0x7f059c004810 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0598001780 tx=0x7f059801c420 comp rx=0 tx=0).stop 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 -- 172.21.15.7:0/1721610603 reap_dead start 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- 172.21.15.7:0/1721610603 shutdown_connections 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- 172.21.15.7:0/1721610603 >> 172.21.15.7:0/1721610603 conn(0x7f05ac06b2c0 msgr2=0x7f05ac06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- 172.21.15.7:0/1721610603 shutdown_connections 2024-03-19T01:24:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- 172.21.15.7:0/1721610603 wait complete. 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 Processor -- start 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- start start 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004020 0x7f05ac13bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac13aa00 0x7f05ac1de010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05ac1de550 0x7f05ac1e1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac151c10 con 0x7f05ac1de550 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac151a90 con 0x7f059c004020 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f05ac151d90 con 0x7f05ac13aa00 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05ac1de550 0x7f05ac1e1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05ac1de550 0x7f05ac1e1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39312/0 (socket says 172.21.15.7:39312) 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 -- 172.21.15.7:0/1708808418 learned_addr learned my addr 172.21.15.7:0/1708808418 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05abfff640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac13aa00 0x7f05ac1de010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b0d8a640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004020 0x7f05ac13bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 -- 172.21.15.7:0/1708808418 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac13aa00 msgr2=0x7f05ac1de010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac13aa00 0x7f05ac1de010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 -- 172.21.15.7:0/1708808418 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004020 msgr2=0x7f05ac13bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004020 0x7f05ac13bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 -- 172.21.15.7:0/1708808418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ac143860 con 0x7f05ac1de550 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05abfff640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac13aa00 0x7f05ac1de010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b158b640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05ac1de550 0x7f05ac1e1580 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f05ac072b40 tx=0x7f05a400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05a97fa640 1 -- 172.21.15.7:0/1708808418 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a400ebc0 con 0x7f05ac1de550 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05a97fa640 1 -- 172.21.15.7:0/1708808418 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a400ed60 con 0x7f05ac1de550 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05a97fa640 1 -- 172.21.15.7:0/1708808418 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a4014670 con 0x7f05ac1de550 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ac143330 con 0x7f05ac1de550 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.342+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ac143050 con 0x7f05ac1de550 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.346+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05ac075570 con 0x7f05ac1de550 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.346+0000 7f05a97fa640 1 -- 172.21.15.7:0/1708808418 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f05a4014810 con 0x7f05ac1de550 2024-03-19T01:24:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.346+0000 7f05a97fa640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0590067690 0x7f0590069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.346+0000 7f05a97fa640 1 -- 172.21.15.7:0/1708808418 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f05a4085640 con 0x7f05ac1de550 2024-03-19T01:24:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.346+0000 7f05b0d8a640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0590067690 0x7f0590069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.346+0000 7f05b0d8a640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0590067690 0x7f0590069b50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f05ac13fad0 tx=0x7f05a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.350+0000 7f05a97fa640 1 -- 172.21.15.7:0/1708808418 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f05ac143330 con 0x7f05ac1de550 2024-03-19T01:24:23.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.506+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f05ac143ad0 con 0x7f05ac1de550 2024-03-19T01:24:23.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.506+0000 7f05a97fa640 1 -- 172.21.15.7:0/1708808418 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) ==== 100+0+11 (secure 0 0 0) 0x7f05a4056350 con 0x7f05ac1de550 2024-03-19T01:24:23.511 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-19T01:24:23.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0590067690 msgr2=0x7f0590069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0590067690 0x7f0590069b50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f05ac13fad0 tx=0x7f05a0008040 comp rx=0 tx=0).stop 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05ac1de550 msgr2=0x7f05ac1e1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05ac1de550 0x7f05ac1e1580 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f05ac072b40 tx=0x7f05a400e530 comp rx=0 tx=0).stop 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 shutdown_connections 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0590067690 0x7f0590069b50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05ac13aa00 0x7f05ac1de010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05ac1de550 0x7f05ac1e1580 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 --2- 172.21.15.7:0/1708808418 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f059c004020 0x7f05ac13bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 >> 172.21.15.7:0/1708808418 conn(0x7f05ac06b2c0 msgr2=0x7f05ac0bb380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 shutdown_connections 2024-03-19T01:24:23.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.510+0000 7f05b2fee640 1 -- 172.21.15.7:0/1708808418 wait complete. 2024-03-19T01:24:23.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:24:23.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3607f64640 1 Processor -- start 2024-03-19T01:24:23.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3607f64640 1 -- start start 2024-03-19T01:24:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3607f64640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3600074390 0x7f3600145e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3607f64640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f360013f5d0 0x7f36000b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3607f64640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36000711c0 con 0x7f36000701a0 2024-03-19T01:24:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3607f64640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f3600071430 con 0x7f360013f5d0 2024-03-19T01:24:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3607f64640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3600074770 con 0x7f3600074390 2024-03-19T01:24:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3606501640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f360013f5d0 0x7f36000b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f36054ff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f36000701a0 0x7f360013eec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35452/0 (socket says 172.21.15.7:35452) 2024-03-19T01:24:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f36054ff640 1 -- 172.21.15.7:0/3001569946 learned_addr learned my addr 172.21.15.7:0/3001569946 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3605d00640 1 --2- 172.21.15.7:0/3001569946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3600074390 0x7f3600145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3606501640 1 -- 172.21.15.7:0/3001569946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3600074390 msgr2=0x7f3600145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3606501640 1 --2- 172.21.15.7:0/3001569946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3600074390 0x7f3600145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3606501640 1 -- 172.21.15.7:0/3001569946 >> v1:172.21.15.7:6789/0 conn(0x7f36000701a0 legacy=0x7f360013eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3606501640 1 -- 172.21.15.7:0/3001569946 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3600145340 con 0x7f360013f5d0 2024-03-19T01:24:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3605d00640 1 --2- 172.21.15.7:0/3001569946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3600074390 0x7f3600145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.650+0000 7f3606501640 1 --2- 172.21.15.7:0/3001569946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f360013f5d0 0x7f36000b6f30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009d90 tx=0x7f35f801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8997e9dc835aad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3604cfe640 1 -- 172.21.15.7:0/3001569946 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f80b2070 con 0x7f360013f5d0 2024-03-19T01:24:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3604cfe640 1 -- 172.21.15.7:0/3001569946 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35f80a5970 con 0x7f360013f5d0 2024-03-19T01:24:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3604cfe640 1 -- 172.21.15.7:0/3001569946 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f80a5d40 con 0x7f360013f5d0 2024-03-19T01:24:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- 172.21.15.7:0/3001569946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f360013f5d0 msgr2=0x7f36000b6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 --2- 172.21.15.7:0/3001569946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f360013f5d0 0x7f36000b6f30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009d90 tx=0x7f35f801c8b0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- 172.21.15.7:0/3001569946 shutdown_connections 2024-03-19T01:24:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 --2- 172.21.15.7:0/3001569946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f360013f5d0 0x7f36000b6f30 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 --2- 172.21.15.7:0/3001569946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3600074390 0x7f3600145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- 172.21.15.7:0/3001569946 >> 172.21.15.7:0/3001569946 conn(0x7f36000b9170 msgr2=0x7f36000b9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- 172.21.15.7:0/3001569946 shutdown_connections 2024-03-19T01:24:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- 172.21.15.7:0/3001569946 wait complete. 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 Processor -- start 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- start start 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36000701a0 0x7f3600132a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3600074390 0x7f36001dd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f360013f5d0 0x7f36001e0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f36001523d0 con 0x7f36000701a0 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f360006fa00 con 0x7f3600074390 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3600152550 con 0x7f360013f5d0 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3605d00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36000701a0 0x7f3600132a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3605d00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36000701a0 0x7f3600132a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39338/0 (socket says 172.21.15.7:39338) 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3605d00640 1 -- 172.21.15.7:0/3445909283 learned_addr learned my addr 172.21.15.7:0/3445909283 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:23.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3606501640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f360013f5d0 0x7f36001e0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f36054ff640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3600074390 0x7f36001dd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3605d00640 1 -- 172.21.15.7:0/3445909283 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f360013f5d0 msgr2=0x7f36001e0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3605d00640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f360013f5d0 0x7f36001e0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3605d00640 1 -- 172.21.15.7:0/3445909283 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3600074390 msgr2=0x7f36001dd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3605d00640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3600074390 0x7f36001dd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3605d00640 1 -- 172.21.15.7:0/3445909283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3600134bb0 con 0x7f36000701a0 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3605d00640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36000701a0 0x7f3600132a80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00c7c0 tx=0x7f35fc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f35f67fc640 1 -- 172.21.15.7:0/3445909283 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc007a30 con 0x7f36000701a0 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f35f67fc640 1 -- 172.21.15.7:0/3445909283 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35fc007bd0 con 0x7f36000701a0 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3600134e90 con 0x7f36000701a0 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f35f67fc640 1 -- 172.21.15.7:0/3445909283 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc0113f0 con 0x7f36000701a0 2024-03-19T01:24:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.654+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3600133d00 con 0x7f36000701a0 2024-03-19T01:24:23.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.658+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3600074a70 con 0x7f36000701a0 2024-03-19T01:24:23.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.658+0000 7f35f67fc640 1 -- 172.21.15.7:0/3445909283 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f35fc011590 con 0x7f36000701a0 2024-03-19T01:24:23.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.658+0000 7f35f67fc640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35e4067660 0x7f35e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:23.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.658+0000 7f36054ff640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35e4067660 0x7f35e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:23.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.658+0000 7f35f67fc640 1 -- 172.21.15.7:0/3445909283 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f35fc085b00 con 0x7f36000701a0 2024-03-19T01:24:23.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.658+0000 7f36054ff640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35e4067660 0x7f35e4069b20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f36001463c0 tx=0x7f35f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:23.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.662+0000 7f35f67fc640 1 -- 172.21.15.7:0/3445909283 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f35fc015020 con 0x7f36000701a0 2024-03-19T01:24:23.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3600075460 con 0x7f36000701a0 2024-03-19T01:24:23.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f35f67fc640 1 -- 172.21.15.7:0/3445909283 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3531 (secure 0 0 0) 0x7f35fc085e20 con 0x7f36000701a0 2024-03-19T01:24:23.860 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:23.860 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4753,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/3744754940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3744754940},{"type":"v1","addr":"172.21.15.167:6833","nonce":3744754940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:24:10.251824+0000","modified":"2024-03-19T01:24:13.280725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5001":{"gid":5001,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/4260183338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4260183338},{"type":"v1","addr":"172.21.15.7:6835","nonce":4260183338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5001,"qdb_cluster":[5001]},"id":2}]} 2024-03-19T01:24:23.861 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 16 2024-03-19T01:24:23.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35e4067660 msgr2=0x7f35e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35e4067660 0x7f35e4069b20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f36001463c0 tx=0x7f35f0009040 comp rx=0 tx=0).stop 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36000701a0 msgr2=0x7f3600132a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36000701a0 0x7f3600132a80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00c7c0 tx=0x7f35fc00cc80 comp rx=0 tx=0).stop 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 shutdown_connections 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f35e4067660 0x7f35e4069b20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f360013f5d0 0x7f36001e0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3600074390 0x7f36001dd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 --2- 172.21.15.7:0/3445909283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36000701a0 0x7f3600132a80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 >> 172.21.15.7:0/3445909283 conn(0x7f36000b9170 msgr2=0x7f360012df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 shutdown_connections 2024-03-19T01:24:23.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.858+0000 7f3607f64640 1 -- 172.21.15.7:0/3445909283 wait complete. 2024-03-19T01:24:23.879 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:23.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b15277640 1 Processor -- start 2024-03-19T01:24:24.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b15277640 1 -- start start 2024-03-19T01:24:24.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b15277640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f1b1009f060 0x7f1b100af380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b15277640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b100aeef0 con 0x7f1b1006bee0 2024-03-19T01:24:24.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b15277640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b1009f440 con 0x7f1b10148340 2024-03-19T01:24:24.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b15277640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f1b1006ba50 con 0x7f1b1009f060 2024-03-19T01:24:24.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0e7fc640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1b1006bee0 0x7f1b10147c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35464/0 (socket says 172.21.15.7:35464) 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0e7fc640 1 -- 172.21.15.7:0/3133248121 learned_addr learned my addr 172.21.15.7:0/3133248121 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0effd640 1 --2- 172.21.15.7:0/3133248121 >> v2:172.21.15.167:3300/0 conn(0x7f1b1009f060 0x7f1b100af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240000306 0 0) 0x7f1b04001120 con 0x7f1b1006bee0 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1af8003530 con 0x7f1b1006bee0 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4033700038 0 0) 0x7f1af8003530 con 0x7f1b1006bee0 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 >> v2:172.21.15.167:3300/0 conn(0x7f1b1009f060 msgr2=0x7f1b100af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 --2- 172.21.15.7:0/3133248121 >> v2:172.21.15.167:3300/0 conn(0x7f1b1009f060 0x7f1b100af380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 >> v1:172.21.15.167:6789/0 conn(0x7f1b10148340 legacy=0x7f1b10151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b100aec10 con 0x7f1b1006bee0 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f1b04001850 con 0x7f1b1006bee0 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 >> v1:172.21.15.7:6789/0 conn(0x7f1b1006bee0 legacy=0x7f1b10147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af80039c0 0x7f1af8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1af8004810 0x7f1af8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1af8005220 0x7f1af800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1006ba50 con 0x7f1af80039c0 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1af800c150 con 0x7f1af8005220 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1af800c350 con 0x7f1af8004810 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0effd640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af80039c0 0x7f1af8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0f7fe640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1af8005220 0x7f1af800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0e7fc640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1af8004810 0x7f1af8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0effd640 1 -- 172.21.15.7:0/3133248121 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1af8004810 msgr2=0x7f1af8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0effd640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1af8004810 0x7f1af8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0effd640 1 -- 172.21.15.7:0/3133248121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1af8005220 msgr2=0x7f1af800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0effd640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1af8005220 0x7f1af800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0effd640 1 -- 172.21.15.7:0/3133248121 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af8003530 con 0x7f1af80039c0 2024-03-19T01:24:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:23.998+0000 7f1b0f7fe640 1 -- 172.21.15.7:0/3133248121 reap_dead start 2024-03-19T01:24:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b0effd640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af80039c0 0x7f1af8003e00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009ae0 tx=0x7f1b0001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bf003192500ac6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b000096b0 con 0x7f1af80039c0 2024-03-19T01:24:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b000a6450 con 0x7f1af80039c0 2024-03-19T01:24:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b0dffb640 1 -- 172.21.15.7:0/3133248121 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b000a6840 con 0x7f1af80039c0 2024-03-19T01:24:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 -- 172.21.15.7:0/3133248121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af80039c0 msgr2=0x7f1af8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af80039c0 0x7f1af8003e00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009ae0 tx=0x7f1b0001ca70 comp rx=0 tx=0).stop 2024-03-19T01:24:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 -- 172.21.15.7:0/3133248121 shutdown_connections 2024-03-19T01:24:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 --2- 172.21.15.7:0/3133248121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af80039c0 0x7f1af8003e00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 -- 172.21.15.7:0/3133248121 >> 172.21.15.7:0/3133248121 conn(0x7f1b100b6510 msgr2=0x7f1b100b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 -- 172.21.15.7:0/3133248121 shutdown_connections 2024-03-19T01:24:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 -- 172.21.15.7:0/3133248121 wait complete. 2024-03-19T01:24:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 Processor -- start 2024-03-19T01:24:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 -- start start 2024-03-19T01:24:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1af80039c0 0x7f1b101d5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b10144e70 0x7f1b101452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b1013bae0 0x7f1b1013bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1006ec00 con 0x7f1b1013bae0 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1006de80 con 0x7f1b10144e70 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.002+0000 7f1b15277640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1b1006ed80 con 0x7f1af80039c0 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b1013bae0 0x7f1b1013bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b1013bae0 0x7f1b1013bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39358/0 (socket says 172.21.15.7:39358) 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0f7fe640 1 -- 172.21.15.7:0/3690174673 learned_addr learned my addr 172.21.15.7:0/3690174673 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0effd640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1af80039c0 0x7f1b101d5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0e7fc640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b10144e70 0x7f1b101452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0f7fe640 1 -- 172.21.15.7:0/3690174673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1af80039c0 msgr2=0x7f1b101d5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0f7fe640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1af80039c0 0x7f1b101d5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0f7fe640 1 -- 172.21.15.7:0/3690174673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b10144e70 msgr2=0x7f1b101452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0f7fe640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b10144e70 0x7f1b101452f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0f7fe640 1 -- 172.21.15.7:0/3690174673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b10143b40 con 0x7f1b1013bae0 2024-03-19T01:24:24.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0f7fe640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b1013bae0 0x7f1b1013bf70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1b100b99e0 tx=0x7f1b0800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:24.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1af77fe640 1 -- 172.21.15.7:0/3690174673 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0801a070 con 0x7f1b1013bae0 2024-03-19T01:24:24.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1af77fe640 1 -- 172.21.15.7:0/3690174673 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b08006d70 con 0x7f1b1013bae0 2024-03-19T01:24:24.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b10146570 con 0x7f1b1013bae0 2024-03-19T01:24:24.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1af77fe640 1 -- 172.21.15.7:0/3690174673 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b08003ba0 con 0x7f1b1013bae0 2024-03-19T01:24:24.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b101417f0 con 0x7f1b1013bae0 2024-03-19T01:24:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b10075460 con 0x7f1b1013bae0 2024-03-19T01:24:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1af77fe640 1 -- 172.21.15.7:0/3690174673 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1b0801d070 con 0x7f1b1013bae0 2024-03-19T01:24:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1af77fe640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1afc067660 0x7f1afc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0effd640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1afc067660 0x7f1afc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1af77fe640 1 -- 172.21.15.7:0/3690174673 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f1b08084cf0 con 0x7f1b1013bae0 2024-03-19T01:24:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.006+0000 7f1b0effd640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1afc067660 0x7f1afc069b20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1b000ad000 tx=0x7f1b000b2040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:24.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.010+0000 7f1af77fe640 1 -- 172.21.15.7:0/3690174673 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1b08089070 con 0x7f1b1013bae0 2024-03-19T01:24:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.202+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b1006c910 con 0x7f1b1013bae0 2024-03-19T01:24:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.202+0000 7f1af77fe640 1 -- 172.21.15.7:0/3690174673 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16633 (secure 0 0 0) 0x7f1b080553b0 con 0x7f1b1013bae0 2024-03-19T01:24:24.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:24.207 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":30,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:24:11.876754+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:24:08.795930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-19T01:24:09.233323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:24.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1afc067660 msgr2=0x7f1afc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1afc067660 0x7f1afc069b20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1b000ad000 tx=0x7f1b000b2040 comp rx=0 tx=0).stop 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b1013bae0 msgr2=0x7f1b1013bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b1013bae0 0x7f1b1013bf70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1b100b99e0 tx=0x7f1b0800fc00 comp rx=0 tx=0).stop 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 shutdown_connections 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f1afc067660 0x7f1afc069b20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b1013bae0 0x7f1b1013bf70 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b10144e70 0x7f1b101452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 --2- 172.21.15.7:0/3690174673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1af80039c0 0x7f1b101d5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 >> 172.21.15.7:0/3690174673 conn(0x7f1b100b6510 msgr2=0x7f1b10071da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 shutdown_connections 2024-03-19T01:24:24.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.206+0000 7f1b15277640 1 -- 172.21.15.7:0/3690174673 wait complete. 2024-03-19T01:24:24.226 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-19T01:24:24.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:24:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.354+0000 7f9084fcd640 1 Processor -- start 2024-03-19T01:24:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.354+0000 7f9084fcd640 1 -- start start 2024-03-19T01:24:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f9084fcd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9080074390 0x7f9080141430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f9084fcd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 0x7f9080151d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f9084fcd640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90800aead0 con 0x7f90800add50 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f9084fcd640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f9080074770 con 0x7f9080074390 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f9084fcd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f90800af050 con 0x7f9080141970 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907e59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9080074390 0x7f9080141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907e59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9080074390 0x7f9080141430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39364/0 (socket says 172.21.15.7:39364) 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907e59c640 1 -- 172.21.15.7:0/3626625711 learned_addr learned my addr 172.21.15.7:0/3626625711 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907ed9d640 1 --2- 172.21.15.7:0/3626625711 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 0x7f9080151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907e59c640 1 -- 172.21.15.7:0/3626625711 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 msgr2=0x7f9080151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907e59c640 1 --2- 172.21.15.7:0/3626625711 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 0x7f9080151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907e59c640 1 -- 172.21.15.7:0/3626625711 >> v1:172.21.15.167:6789/0 conn(0x7f90800add50 legacy=0x7f9080147830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907e59c640 1 -- 172.21.15.7:0/3626625711 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90800aed70 con 0x7f9080074390 2024-03-19T01:24:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907ed9d640 1 --2- 172.21.15.7:0/3626625711 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 0x7f9080151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907e59c640 1 --2- 172.21.15.7:0/3626625711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080074390 0x7f9080141430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f90800b68c0 tx=0x7f907001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0f509c26add3a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907d59a640 1 -- 172.21.15.7:0/3626625711 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90700b0070 con 0x7f9080074390 2024-03-19T01:24:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907d59a640 1 -- 172.21.15.7:0/3626625711 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90700ab600 con 0x7f9080074390 2024-03-19T01:24:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.358+0000 7f907d59a640 1 -- 172.21.15.7:0/3626625711 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90700ab9d0 con 0x7f9080074390 2024-03-19T01:24:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- 172.21.15.7:0/3626625711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080074390 msgr2=0x7f9080141430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 --2- 172.21.15.7:0/3626625711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080074390 0x7f9080141430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f90800b68c0 tx=0x7f907001c3e0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- 172.21.15.7:0/3626625711 shutdown_connections 2024-03-19T01:24:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 --2- 172.21.15.7:0/3626625711 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 0x7f9080151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 --2- 172.21.15.7:0/3626625711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080074390 0x7f9080141430 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- 172.21.15.7:0/3626625711 >> 172.21.15.7:0/3626625711 conn(0x7f90800b5bf0 msgr2=0x7f90800b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- 172.21.15.7:0/3626625711 shutdown_connections 2024-03-19T01:24:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- 172.21.15.7:0/3626625711 wait complete. 2024-03-19T01:24:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 Processor -- start 2024-03-19T01:24:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- start start 2024-03-19T01:24:24.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90800add50 0x7f9080133240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 0x7f908012d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080143230 0x7f908012da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9080154c40 con 0x7f9080143230 2024-03-19T01:24:24.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9080154ac0 con 0x7f90800add50 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9080154dc0 con 0x7f9080141970 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080143230 0x7f908012da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080143230 0x7f908012da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39370/0 (socket says 172.21.15.7:39370) 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907ed9d640 1 -- 172.21.15.7:0/3380419491 learned_addr learned my addr 172.21.15.7:0/3380419491 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907dd9b640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 0x7f908012d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907e59c640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90800add50 0x7f9080133240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907ed9d640 1 -- 172.21.15.7:0/3380419491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 msgr2=0x7f908012d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907ed9d640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 0x7f908012d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907ed9d640 1 -- 172.21.15.7:0/3380419491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90800add50 msgr2=0x7f9080133240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907ed9d640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90800add50 0x7f9080133240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907ed9d640 1 -- 172.21.15.7:0/3380419491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90801436f0 con 0x7f9080143230 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f907ed9d640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080143230 0x7f908012da10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f907800cd80 tx=0x7f907800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:24.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f906effd640 1 -- 172.21.15.7:0/3380419491 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907800f680 con 0x7f9080143230 2024-03-19T01:24:24.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f906effd640 1 -- 172.21.15.7:0/3380419491 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f907800f840 con 0x7f9080143230 2024-03-19T01:24:24.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9080141fe0 con 0x7f9080143230 2024-03-19T01:24:24.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f906effd640 1 -- 172.21.15.7:0/3380419491 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907800fc30 con 0x7f9080143230 2024-03-19T01:24:24.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.362+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90800b19a0 con 0x7f9080143230 2024-03-19T01:24:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.366+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90801425f0 con 0x7f9080143230 2024-03-19T01:24:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.366+0000 7f906effd640 1 -- 172.21.15.7:0/3380419491 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f90780213f0 con 0x7f9080143230 2024-03-19T01:24:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.366+0000 7f906effd640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9060067660 0x7f9060069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.366+0000 7f906effd640 1 -- 172.21.15.7:0/3380419491 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f90780865a0 con 0x7f9080143230 2024-03-19T01:24:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.366+0000 7f907e59c640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9060067660 0x7f9060069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.366+0000 7f907e59c640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9060067660 0x7f9060069b20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f90700ad000 tx=0x7f9070008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:24.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.374+0000 7f906effd640 1 -- 172.21.15.7:0/3380419491 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9078014030 con 0x7f9080143230 2024-03-19T01:24:24.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.570+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9080075460 con 0x7f9080143230 2024-03-19T01:24:24.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.570+0000 7f906effd640 1 -- 172.21.15.7:0/3380419491 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3531 (secure 0 0 0) 0x7f90780572b0 con 0x7f9080143230 2024-03-19T01:24:24.575 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4753,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/3744754940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3744754940},{"type":"v1","addr":"172.21.15.167:6833","nonce":3744754940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:24:10.251824+0000","modified":"2024-03-19T01:24:13.280725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5001":{"gid":5001,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/4260183338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4260183338},{"type":"v1","addr":"172.21.15.7:6835","nonce":4260183338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5001,"qdb_cluster":[5001]},"id":2}]} 2024-03-19T01:24:24.575 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 16 2024-03-19T01:24:24.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9060067660 msgr2=0x7f9060069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9060067660 0x7f9060069b20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f90700ad000 tx=0x7f9070008040 comp rx=0 tx=0).stop 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080143230 msgr2=0x7f908012da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080143230 0x7f908012da10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f907800cd80 tx=0x7f907800a650 comp rx=0 tx=0).stop 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 shutdown_connections 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9060067660 0x7f9060069b20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9080141970 0x7f908012d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9080143230 0x7f908012da10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 --2- 172.21.15.7:0/3380419491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90800add50 0x7f9080133240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 >> 172.21.15.7:0/3380419491 conn(0x7f90800b5bf0 msgr2=0x7f9080067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 shutdown_connections 2024-03-19T01:24:24.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.574+0000 7f9084fcd640 1 -- 172.21.15.7:0/3380419491 wait complete. 2024-03-19T01:24:24.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:24:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec25e17640 1 Processor -- start 2024-03-19T01:24:24.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec25e17640 1 -- start start 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec25e17640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fec20072fa0 0x7fec200b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec25e17640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fec2013efd0 0x7fec2013f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec25e17640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec20073380 con 0x7fec20145f10 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec25e17640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fec20145430 con 0x7fec2013efd0 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec25e17640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fec20145710 con 0x7fec20072fa0 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1effd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fec20145f10 0x7fec20071550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35480/0 (socket says 172.21.15.7:35480) 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1effd640 1 -- 172.21.15.7:0/790927073 learned_addr learned my addr 172.21.15.7:0/790927073 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1ffff640 1 --2- 172.21.15.7:0/790927073 >> v2:172.21.15.7:3300/0 conn(0x7fec2013efd0 0x7fec2013f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1f7fe640 1 --2- 172.21.15.7:0/790927073 >> v2:172.21.15.167:3300/0 conn(0x7fec20072fa0 0x7fec200b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1e7fc640 1 -- 172.21.15.7:0/790927073 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3129818750 0 0) 0x7fec20073380 con 0x7fec20145f10 2024-03-19T01:24:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1e7fc640 1 -- 172.21.15.7:0/790927073 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec08003530 con 0x7fec20145f10 2024-03-19T01:24:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1ffff640 1 -- 172.21.15.7:0/790927073 >> v2:172.21.15.167:3300/0 conn(0x7fec20072fa0 msgr2=0x7fec200b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1ffff640 1 --2- 172.21.15.7:0/790927073 >> v2:172.21.15.167:3300/0 conn(0x7fec20072fa0 0x7fec200b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1ffff640 1 -- 172.21.15.7:0/790927073 >> v1:172.21.15.7:6789/0 conn(0x7fec20145f10 legacy=0x7fec20071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1ffff640 1 -- 172.21.15.7:0/790927073 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec201459f0 con 0x7fec2013efd0 2024-03-19T01:24:24.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1f7fe640 1 --2- 172.21.15.7:0/790927073 >> v2:172.21.15.167:3300/0 conn(0x7fec20072fa0 0x7fec200b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:24.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.706+0000 7fec1ffff640 1 --2- 172.21.15.7:0/790927073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec2013efd0 0x7fec2013f410 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fec200bb880 tx=0x7fec1801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2e21282a4fcc64 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:24.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1e7fc640 1 -- 172.21.15.7:0/790927073 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec180a66e0 con 0x7fec2013efd0 2024-03-19T01:24:24.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1e7fc640 1 -- 172.21.15.7:0/790927073 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec180a6880 con 0x7fec2013efd0 2024-03-19T01:24:24.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1e7fc640 1 -- 172.21.15.7:0/790927073 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec180a6c50 con 0x7fec2013efd0 2024-03-19T01:24:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- 172.21.15.7:0/790927073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec2013efd0 msgr2=0x7fec2013f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 --2- 172.21.15.7:0/790927073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec2013efd0 0x7fec2013f410 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fec200bb880 tx=0x7fec1801c810 comp rx=0 tx=0).stop 2024-03-19T01:24:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- 172.21.15.7:0/790927073 shutdown_connections 2024-03-19T01:24:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 --2- 172.21.15.7:0/790927073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec2013efd0 0x7fec2013f410 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 --2- 172.21.15.7:0/790927073 >> v2:172.21.15.167:3300/0 conn(0x7fec20072fa0 0x7fec200b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- 172.21.15.7:0/790927073 >> 172.21.15.7:0/790927073 conn(0x7fec2006bc10 msgr2=0x7fec200bac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:24.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- 172.21.15.7:0/790927073 shutdown_connections 2024-03-19T01:24:24.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- 172.21.15.7:0/790927073 wait complete. 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 Processor -- start 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- start start 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec20072fa0 0x7fec2006d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec2013efd0 0x7fec2006c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec20145f10 0x7fec200b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fec20152460 con 0x7fec20145f10 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fec201522e0 con 0x7fec2013efd0 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fec201525e0 con 0x7fec20072fa0 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec20145f10 0x7fec200b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec20145f10 0x7fec200b46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39378/0 (socket says 172.21.15.7:39378) 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1ffff640 1 -- 172.21.15.7:0/1202856283 learned_addr learned my addr 172.21.15.7:0/1202856283 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:24.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1effd640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec2013efd0 0x7fec2006c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1f7fe640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec20072fa0 0x7fec2006d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1ffff640 1 -- 172.21.15.7:0/1202856283 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec20072fa0 msgr2=0x7fec2006d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1ffff640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec20072fa0 0x7fec2006d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1ffff640 1 -- 172.21.15.7:0/1202856283 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec2013efd0 msgr2=0x7fec2006c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1ffff640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec2013efd0 0x7fec2006c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1ffff640 1 -- 172.21.15.7:0/1202856283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec2006de50 con 0x7fec20145f10 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec1ffff640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec20145f10 0x7fec200b46e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fec18019420 tx=0x7fec180195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec07fff640 1 -- 172.21.15.7:0/1202856283 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec180be070 con 0x7fec20145f10 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec07fff640 1 -- 172.21.15.7:0/1202856283 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec180b8630 con 0x7fec20145f10 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec200b4200 con 0x7fec20145f10 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec07fff640 1 -- 172.21.15.7:0/1202856283 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec180b8a20 con 0x7fec20145f10 2024-03-19T01:24:24.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.710+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec2006ccd0 con 0x7fec20145f10 2024-03-19T01:24:24.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.714+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec200b43d0 con 0x7fec20145f10 2024-03-19T01:24:24.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.714+0000 7fec07fff640 1 -- 172.21.15.7:0/1202856283 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fec180c1070 con 0x7fec20145f10 2024-03-19T01:24:24.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.714+0000 7fec07fff640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec0c067660 0x7fec0c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:24.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.714+0000 7fec07fff640 1 -- 172.21.15.7:0/1202856283 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fec18120b80 con 0x7fec20145f10 2024-03-19T01:24:24.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.714+0000 7fec1f7fe640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec0c067660 0x7fec0c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:24.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.714+0000 7fec1f7fe640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec0c067660 0x7fec0c069b20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fec1000b930 tx=0x7fec10008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:24.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:24.718+0000 7fec07fff640 1 -- 172.21.15.7:0/1202856283 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fec2006de50 con 0x7fec20145f10 2024-03-19T01:24:25.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec200bac20 con 0x7fec20145f10 2024-03-19T01:24:25.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec07fff640 1 -- 172.21.15.7:0/1202856283 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3531 (secure 0 0 0) 0x7fec18016520 con 0x7fec20145f10 2024-03-19T01:24:25.108 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:25.108 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4753,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/3744754940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3744754940},{"type":"v1","addr":"172.21.15.167:6833","nonce":3744754940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:24:10.251824+0000","modified":"2024-03-19T01:24:13.280725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5001":{"gid":5001,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/4260183338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4260183338},{"type":"v1","addr":"172.21.15.7:6835","nonce":4260183338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5001,"qdb_cluster":[5001]},"id":2}]} 2024-03-19T01:24:25.109 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 16 2024-03-19T01:24:25.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec0c067660 msgr2=0x7fec0c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec0c067660 0x7fec0c069b20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fec1000b930 tx=0x7fec10008040 comp rx=0 tx=0).stop 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec20145f10 msgr2=0x7fec200b46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec20145f10 0x7fec200b46e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fec18019420 tx=0x7fec180195b0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 shutdown_connections 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fec0c067660 0x7fec0c069b20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec20145f10 0x7fec200b46e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec2013efd0 0x7fec2006c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 --2- 172.21.15.7:0/1202856283 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec20072fa0 0x7fec2006d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 >> 172.21.15.7:0/1202856283 conn(0x7fec2006bc10 msgr2=0x7fec201447f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 shutdown_connections 2024-03-19T01:24:25.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.106+0000 7fec25e17640 1 -- 172.21.15.7:0/1202856283 wait complete. 2024-03-19T01:24:25.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.234+0000 7f87a7881640 1 Processor -- start 2024-03-19T01:24:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.234+0000 7f87a7881640 1 -- start start 2024-03-19T01:24:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f87a0148420 0x7f87a006e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a00af510 0x7f87a00af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87a00aec10 con 0x7f87a006df50 2024-03-19T01:24:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f87a006be90 con 0x7f87a0148420 2024-03-19T01:24:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f87a009f440 con 0x7f87a00af510 2024-03-19T01:24:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a4e1c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f87a0148420 0x7f87a006e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a4e1c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f87a0148420 0x7f87a006e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39384/0 (socket says 172.21.15.7:39384) 2024-03-19T01:24:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a561d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f87a006df50 0x7f87a0147d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35494/0 (socket says 172.21.15.7:35494) 2024-03-19T01:24:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a4e1c640 1 -- 172.21.15.7:0/1893311399 learned_addr learned my addr 172.21.15.7:0/1893311399 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a5e1e640 1 --2- 172.21.15.7:0/1893311399 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a00af510 0x7f87a00af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a4e1c640 1 -- 172.21.15.7:0/1893311399 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a00af510 msgr2=0x7f87a00af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a4e1c640 1 --2- 172.21.15.7:0/1893311399 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a00af510 0x7f87a00af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a4e1c640 1 -- 172.21.15.7:0/1893311399 >> v1:172.21.15.7:6789/0 conn(0x7f87a006df50 legacy=0x7f87a0147d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a4e1c640 1 -- 172.21.15.7:0/1893311399 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a006c0e0 con 0x7f87a0148420 2024-03-19T01:24:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f8797fff640 1 -- 172.21.15.7:0/1893311399 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2472240325 0 0) 0x7f87a00aec10 con 0x7f87a006df50 2024-03-19T01:24:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a4e1c640 1 --2- 172.21.15.7:0/1893311399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a0148420 0x7f87a006e9e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8790009870 tx=0x7f879001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1c56ab208d3bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f8797fff640 1 -- 172.21.15.7:0/1893311399 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87900b0070 con 0x7f87a0148420 2024-03-19T01:24:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f8797fff640 1 -- 172.21.15.7:0/1893311399 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8790004230 con 0x7f87a0148420 2024-03-19T01:24:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f8797fff640 1 -- 172.21.15.7:0/1893311399 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790003620 con 0x7f87a0148420 2024-03-19T01:24:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 -- 172.21.15.7:0/1893311399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a0148420 msgr2=0x7f87a006e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 --2- 172.21.15.7:0/1893311399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a0148420 0x7f87a006e9e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8790009870 tx=0x7f879001c440 comp rx=0 tx=0).stop 2024-03-19T01:24:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 -- 172.21.15.7:0/1893311399 shutdown_connections 2024-03-19T01:24:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 --2- 172.21.15.7:0/1893311399 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a00af510 0x7f87a00af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 --2- 172.21.15.7:0/1893311399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a0148420 0x7f87a006e9e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 -- 172.21.15.7:0/1893311399 >> 172.21.15.7:0/1893311399 conn(0x7f87a00b6510 msgr2=0x7f87a00b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 -- 172.21.15.7:0/1893311399 shutdown_connections 2024-03-19T01:24:25.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.238+0000 7f87a7881640 1 -- 172.21.15.7:0/1893311399 wait complete. 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 Processor -- start 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 -- start start 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a006df50 0x7f87a0144920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87a00af510 0x7f87a0142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a0148420 0x7f87a01dff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f87a0154730 con 0x7f87a006df50 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f87a01545b0 con 0x7f87a00af510 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f87a01548b0 con 0x7f87a0148420 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a561d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a006df50 0x7f87a0144920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a561d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a006df50 0x7f87a0144920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39392/0 (socket says 172.21.15.7:39392) 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a561d640 1 -- 172.21.15.7:0/2460400943 learned_addr learned my addr 172.21.15.7:0/2460400943 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a4e1c640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87a00af510 0x7f87a0142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a5e1e640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a0148420 0x7f87a01dff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a561d640 1 -- 172.21.15.7:0/2460400943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a0148420 msgr2=0x7f87a01dff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a561d640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a0148420 0x7f87a01dff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a561d640 1 -- 172.21.15.7:0/2460400943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87a00af510 msgr2=0x7f87a0142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a561d640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87a00af510 0x7f87a0142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a561d640 1 -- 172.21.15.7:0/2460400943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a01441a0 con 0x7f87a006df50 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a4e1c640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87a00af510 0x7f87a0142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a561d640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a006df50 0x7f87a0144920 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f87980011f0 tx=0x7f879800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f8795ffb640 1 -- 172.21.15.7:0/2460400943 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798018070 con 0x7f87a006df50 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f8795ffb640 1 -- 172.21.15.7:0/2460400943 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87980074e0 con 0x7f87a006df50 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a00b3890 con 0x7f87a006df50 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f8795ffb640 1 -- 172.21.15.7:0/2460400943 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798004a10 con 0x7f87a006df50 2024-03-19T01:24:25.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a0143ec0 con 0x7f87a006df50 2024-03-19T01:24:25.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a006c350 con 0x7f87a006df50 2024-03-19T01:24:25.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f8795ffb640 1 -- 172.21.15.7:0/2460400943 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f879801b070 con 0x7f87a006df50 2024-03-19T01:24:25.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f8795ffb640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8784067660 0x7f8784069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f8795ffb640 1 -- 172.21.15.7:0/2460400943 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f8798084f80 con 0x7f87a006df50 2024-03-19T01:24:25.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.242+0000 7f87a4e1c640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8784067660 0x7f8784069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.246+0000 7f87a4e1c640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8784067660 0x7f8784069b20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f87900ad000 tx=0x7f87900a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:25.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.250+0000 7f8795ffb640 1 -- 172.21.15.7:0/2460400943 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8798055c70 con 0x7f87a006df50 2024-03-19T01:24:25.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.406+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87a00b5dc0 con 0x7f87a006df50 2024-03-19T01:24:25.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f8795ffb640 1 -- 172.21.15.7:0/2460400943 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16633 (secure 0 0 0) 0x7f8798055570 con 0x7f87a006df50 2024-03-19T01:24:25.412 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:25.412 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":30,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:24:11.876754+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:24:08.795930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-19T01:24:09.233323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:25.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8784067660 msgr2=0x7f8784069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8784067660 0x7f8784069b20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f87900ad000 tx=0x7f87900a6000 comp rx=0 tx=0).stop 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a006df50 msgr2=0x7f87a0144920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a006df50 0x7f87a0144920 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f87980011f0 tx=0x7f879800c9b0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 shutdown_connections 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8784067660 0x7f8784069b20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87a0148420 0x7f87a01dff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87a00af510 0x7f87a0142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 --2- 172.21.15.7:0/2460400943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87a006df50 0x7f87a0144920 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 >> 172.21.15.7:0/2460400943 conn(0x7f87a00b6510 msgr2=0x7f87a01410d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 shutdown_connections 2024-03-19T01:24:25.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.410+0000 7f87a7881640 1 -- 172.21.15.7:0/2460400943 wait complete. 2024-03-19T01:24:25.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-19T01:24:25.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8c482640 1 Processor -- start 2024-03-19T01:24:25.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8c482640 1 -- start start 2024-03-19T01:24:25.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8c482640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f7e84074390 0x7f7e84141430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8c482640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e84141970 0x7f7e84151d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8c482640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e840aead0 con 0x7f7e840add50 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8c482640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f7e84074770 con 0x7f7e84141970 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8c482640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f7e840af050 con 0x7f7e84074390 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8aa1f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e84141970 0x7f7e84151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e89a1d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7e840add50 0x7f7e84147830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35504/0 (socket says 172.21.15.7:35504) 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e89a1d640 1 -- 172.21.15.7:0/685362043 learned_addr learned my addr 172.21.15.7:0/685362043 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8aa1f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e84141970 0x7f7e84151d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39400/0 (socket says 172.21.15.7:39400) 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8a21e640 1 --2- 172.21.15.7:0/685362043 >> v2:172.21.15.167:3300/0 conn(0x7f7e84074390 0x7f7e84141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8aa1f640 1 -- 172.21.15.7:0/685362043 >> v2:172.21.15.167:3300/0 conn(0x7f7e84074390 msgr2=0x7f7e84141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8aa1f640 1 --2- 172.21.15.7:0/685362043 >> v2:172.21.15.167:3300/0 conn(0x7f7e84074390 0x7f7e84141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8aa1f640 1 -- 172.21.15.7:0/685362043 >> v1:172.21.15.7:6789/0 conn(0x7f7e840add50 legacy=0x7f7e84147830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.538+0000 7f7e8aa1f640 1 -- 172.21.15.7:0/685362043 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e840aed70 con 0x7f7e84141970 2024-03-19T01:24:25.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 --2- 172.21.15.7:0/685362043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 0x7f7e84151d60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7e840b68c0 tx=0x7f7e7c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fda034404b2cab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:25.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8921c640 1 -- 172.21.15.7:0/685362043 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0b1070 con 0x7f7e84141970 2024-03-19T01:24:25.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8921c640 1 -- 172.21.15.7:0/685362043 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e7c0a4b70 con 0x7f7e84141970 2024-03-19T01:24:25.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8921c640 1 -- 172.21.15.7:0/685362043 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0ac440 con 0x7f7e84141970 2024-03-19T01:24:25.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- 172.21.15.7:0/685362043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 msgr2=0x7f7e84151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 --2- 172.21.15.7:0/685362043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 0x7f7e84151d60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7e840b68c0 tx=0x7f7e7c01c3b0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- 172.21.15.7:0/685362043 shutdown_connections 2024-03-19T01:24:25.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 --2- 172.21.15.7:0/685362043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 0x7f7e84151d60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 --2- 172.21.15.7:0/685362043 >> v2:172.21.15.167:3300/0 conn(0x7f7e84074390 0x7f7e84141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- 172.21.15.7:0/685362043 >> 172.21.15.7:0/685362043 conn(0x7f7e840b5bf0 msgr2=0x7f7e840b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:25.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- 172.21.15.7:0/685362043 shutdown_connections 2024-03-19T01:24:25.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- 172.21.15.7:0/685362043 wait complete. 2024-03-19T01:24:25.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 Processor -- start 2024-03-19T01:24:25.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- start start 2024-03-19T01:24:25.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e84074390 0x7f7e841e4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e840add50 0x7f7e841374c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 0x7f7e84137a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7e84155670 con 0x7f7e84141970 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7e841554f0 con 0x7f7e840add50 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7e841557f0 con 0x7f7e84074390 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 0x7f7e84137a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 0x7f7e84137a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39414/0 (socket says 172.21.15.7:39414) 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 -- 172.21.15.7:0/3265921741 learned_addr learned my addr 172.21.15.7:0/3265921741 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 -- 172.21.15.7:0/3265921741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e84074390 msgr2=0x7f7e841e4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e84074390 0x7f7e841e4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 -- 172.21.15.7:0/3265921741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e840add50 msgr2=0x7f7e841374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e840add50 0x7f7e841374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 -- 172.21.15.7:0/3265921741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e84136700 con 0x7f7e84141970 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8aa1f640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 0x7f7e84137a00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0af040 tx=0x7f7e7c0194d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e76ffd640 1 -- 172.21.15.7:0/3265921741 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0b1050 con 0x7f7e84141970 2024-03-19T01:24:25.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e76ffd640 1 -- 172.21.15.7:0/3265921741 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e7c0bd070 con 0x7f7e84141970 2024-03-19T01:24:25.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e84136a40 con 0x7f7e84141970 2024-03-19T01:24:25.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.542+0000 7f7e76ffd640 1 -- 172.21.15.7:0/3265921741 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c009450 con 0x7f7e84141970 2024-03-19T01:24:25.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.546+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e841387a0 con 0x7f7e84141970 2024-03-19T01:24:25.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.546+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e84075460 con 0x7f7e84141970 2024-03-19T01:24:25.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.546+0000 7f7e76ffd640 1 -- 172.21.15.7:0/3265921741 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7e7c0b8690 con 0x7f7e84141970 2024-03-19T01:24:25.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.546+0000 7f7e76ffd640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e68067660 0x7f7e68069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.546+0000 7f7e76ffd640 1 -- 172.21.15.7:0/3265921741 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f7e7c120000 con 0x7f7e84141970 2024-03-19T01:24:25.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.546+0000 7f7e8a21e640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e68067660 0x7f7e68069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.546+0000 7f7e8a21e640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e68067660 0x7f7e68069b20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7e800098f0 tx=0x7f7e80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:25.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.554+0000 7f7e76ffd640 1 -- 172.21.15.7:0/3265921741 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7e84136a40 con 0x7f7e84141970 2024-03-19T01:24:25.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.742+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7e84137170 con 0x7f7e84141970 2024-03-19T01:24:25.747 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-19T01:24:25.746+0000 7fd4c7382640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T01:24:25.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.746+0000 7f7e76ffd640 1 -- 172.21.15.7:0/3265921741 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f7e7c016440 con 0x7f7e84141970 2024-03-19T01:24:25.749 INFO:teuthology.orchestra.run.smithi007.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-19T01:24:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e68067660 msgr2=0x7f7e68069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e68067660 0x7f7e68069b20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7e800098f0 tx=0x7f7e80008040 comp rx=0 tx=0).stop 2024-03-19T01:24:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 msgr2=0x7f7e84137a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 0x7f7e84137a00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0af040 tx=0x7f7e7c0194d0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 shutdown_connections 2024-03-19T01:24:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e68067660 0x7f7e68069b20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84141970 0x7f7e84137a00 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e840add50 0x7f7e841374c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 --2- 172.21.15.7:0/3265921741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e84074390 0x7f7e841e4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 >> 172.21.15.7:0/3265921741 conn(0x7f7e840b5bf0 msgr2=0x7f7e84067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 shutdown_connections 2024-03-19T01:24:25.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.750+0000 7f7e8c482640 1 -- 172.21.15.7:0/3265921741 wait complete. 2024-03-19T01:24:25.754 INFO:tasks.ceph.mds.a.smithi007.stderr: -2410> 2024-03-19T01:24:15.850+0000 7f7c142e6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T01:24:25.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-19T01:24:25.827 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2024-03-19T01:24:25.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f66402e0640 1 Processor -- start 2024-03-19T01:24:25.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f66402e0640 1 -- start start 2024-03-19T01:24:25.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f66402e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66380b4eb0 0x7f66381477d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f66402e0640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f6638147d10 0x7f663813de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f66402e0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66381481a0 con 0x7f66380b1600 2024-03-19T01:24:25.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f66402e0640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f66380b5e30 con 0x7f6638147d10 2024-03-19T01:24:25.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f66402e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f66380b23b0 con 0x7f66380b4eb0 2024-03-19T01:24:25.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f663e87d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f6638147d10 0x7f663813de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f663e07c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f66380b1600 0x7f66380b1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:55342/0 (socket says 172.21.15.7:55342) 2024-03-19T01:24:25.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f663e07c640 1 -- 172.21.15.7:0/3923308804 learned_addr learned my addr 172.21.15.7:0/3923308804 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:25.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f663d87b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66380b4eb0 0x7f66381477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f663e87d640 1 -- 172.21.15.7:0/3923308804 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66380b4eb0 msgr2=0x7f66381477d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:25.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f663e87d640 1 --2- 172.21.15.7:0/3923308804 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66380b4eb0 0x7f66381477d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f663e87d640 1 -- 172.21.15.7:0/3923308804 >> v1:172.21.15.167:6789/0 conn(0x7f66380b1600 legacy=0x7f66380b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.942+0000 7f663e87d640 1 -- 172.21.15.7:0/3923308804 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66380b19e0 con 0x7f6638147d10 2024-03-19T01:24:25.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663e87d640 1 --2- 172.21.15.7:0/3923308804 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6638147d10 0x7f663813de30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6630009d90 tx=0x7f663001c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=362890b069c4df5f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:25.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d07a640 1 -- 172.21.15.7:0/3923308804 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66300b2070 con 0x7f6638147d10 2024-03-19T01:24:25.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d07a640 1 -- 172.21.15.7:0/3923308804 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66300a5890 con 0x7f6638147d10 2024-03-19T01:24:25.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d07a640 1 -- 172.21.15.7:0/3923308804 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66300a5c60 con 0x7f6638147d10 2024-03-19T01:24:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 -- 172.21.15.7:0/3923308804 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6638147d10 msgr2=0x7f663813de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 --2- 172.21.15.7:0/3923308804 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6638147d10 0x7f663813de30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6630009d90 tx=0x7f663001c8e0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 -- 172.21.15.7:0/3923308804 shutdown_connections 2024-03-19T01:24:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 --2- 172.21.15.7:0/3923308804 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6638147d10 0x7f663813de30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 --2- 172.21.15.7:0/3923308804 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66380b4eb0 0x7f66381477d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 -- 172.21.15.7:0/3923308804 >> 172.21.15.7:0/3923308804 conn(0x7f663806c3b0 msgr2=0x7f663806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 -- 172.21.15.7:0/3923308804 shutdown_connections 2024-03-19T01:24:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 -- 172.21.15.7:0/3923308804 wait complete. 2024-03-19T01:24:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 Processor -- start 2024-03-19T01:24:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 -- start start 2024-03-19T01:24:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66380b1600 0x7f6638140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66380b4eb0 0x7f66381403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6638147d10 0x7f6638070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f66380b3220 con 0x7f66380b4eb0 2024-03-19T01:24:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f663813e840 con 0x7f66380b1600 2024-03-19T01:24:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f66402e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f66380b33a0 con 0x7f6638147d10 2024-03-19T01:24:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d87b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66380b4eb0 0x7f66381403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d87b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66380b4eb0 0x7f66381403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39446/0 (socket says 172.21.15.7:39446) 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d87b640 1 -- 172.21.15.7:0/3779106898 learned_addr learned my addr 172.21.15.7:0/3779106898 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663e87d640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6638147d10 0x7f6638070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663e07c640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66380b1600 0x7f6638140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d87b640 1 -- 172.21.15.7:0/3779106898 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6638147d10 msgr2=0x7f6638070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d87b640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6638147d10 0x7f6638070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d87b640 1 -- 172.21.15.7:0/3779106898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66380b1600 msgr2=0x7f6638140ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d87b640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66380b1600 0x7f6638140ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663d87b640 1 -- 172.21.15.7:0/3779106898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663813f350 con 0x7f66380b4eb0 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.946+0000 7f663e07c640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66380b1600 0x7f6638140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f663d87b640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66380b4eb0 0x7f66381403f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f662800e960 tx=0x7f662800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f662effd640 1 -- 172.21.15.7:0/3779106898 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628017070 con 0x7f66380b4eb0 2024-03-19T01:24:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66381418c0 con 0x7f66380b4eb0 2024-03-19T01:24:25.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f662effd640 1 -- 172.21.15.7:0/3779106898 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f662800ca00 con 0x7f66380b4eb0 2024-03-19T01:24:25.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f662effd640 1 -- 172.21.15.7:0/3779106898 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662800cdb0 con 0x7f66380b4eb0 2024-03-19T01:24:25.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6638140930 con 0x7f66380b4eb0 2024-03-19T01:24:25.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6638075570 con 0x7f66380b4eb0 2024-03-19T01:24:25.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f662effd640 1 -- 172.21.15.7:0/3779106898 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f662801a070 con 0x7f66380b4eb0 2024-03-19T01:24:25.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f662effd640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f661c067660 0x7f661c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:25.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f662effd640 1 -- 172.21.15.7:0/3779106898 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5795+0+0 (secure 0 0 0) 0x7f6628085950 con 0x7f66380b4eb0 2024-03-19T01:24:25.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f663e07c640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f661c067660 0x7f661c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:25.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.950+0000 7f663e07c640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f661c067660 0x7f661c069b20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6634007fd0 tx=0x7f6634009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:25.954+0000 7f662effd640 1 -- 172.21.15.7:0/3779106898 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6628056580 con 0x7f66380b4eb0 2024-03-19T01:24:26.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.142+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6638144ff0 con 0x7f66380b4eb0 2024-03-19T01:24:26.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f662effd640 1 -- 172.21.15.7:0/3779106898 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7f6628055e80 con 0x7f66380b4eb0 2024-03-19T01:24:26.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f661c067660 msgr2=0x7f661c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f66402e0640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f661c067660 0x7f661c069b20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6634007fd0 tx=0x7f6634009040 comp rx=0 tx=0).stop 2024-03-19T01:24:26.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66380b4eb0 msgr2=0x7f66381403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f66402e0640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66380b4eb0 0x7f66381403f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f662800e960 tx=0x7f662800ee20 comp rx=0 tx=0).stop 2024-03-19T01:24:26.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 shutdown_connections 2024-03-19T01:24:26.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f66402e0640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f661c067660 0x7f661c069b20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f66402e0640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6638147d10 0x7f6638070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f66402e0640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66380b4eb0 0x7f66381403f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.750+0000 7f66402e0640 1 --2- 172.21.15.7:0/3779106898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66380b1600 0x7f6638140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.754+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 >> 172.21.15.7:0/3779106898 conn(0x7f663806c3b0 msgr2=0x7f6638144120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:26.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.754+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 shutdown_connections 2024-03-19T01:24:26.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.754+0000 7f66402e0640 1 -- 172.21.15.7:0/3779106898 wait complete. 2024-03-19T01:24:26.776 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-19T01:24:26.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306f649640 1 Processor -- start 2024-03-19T01:24:26.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306f649640 1 -- start start 2024-03-19T01:24:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306f649640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b6380 0x7f30680b7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306f649640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30680b7c60 0x7f30681414a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306f649640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30680b6760 con 0x7f30681419e0 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306f649640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f30680b6ba0 con 0x7f30680b7c60 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306f649640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f30680b6e80 con 0x7f30680b6380 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306cbe4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30680b7c60 0x7f30681414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306cbe4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30680b7c60 0x7f30681414a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39448/0 (socket says 172.21.15.7:39448) 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306cbe4640 1 -- 172.21.15.7:0/3496119636 learned_addr learned my addr 172.21.15.7:0/3496119636 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.906+0000 7f306d3e5640 1 --2- 172.21.15.7:0/3496119636 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b6380 0x7f30680b7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306cbe4640 1 -- 172.21.15.7:0/3496119636 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b6380 msgr2=0x7f30680b7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306cbe4640 1 --2- 172.21.15.7:0/3496119636 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b6380 0x7f30680b7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306cbe4640 1 -- 172.21.15.7:0/3496119636 >> v1:172.21.15.167:6789/0 conn(0x7f30681419e0 legacy=0x7f3068071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306cbe4640 1 -- 172.21.15.7:0/3496119636 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30680b5e20 con 0x7f30680b7c60 2024-03-19T01:24:26.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306cbe4640 1 --2- 172.21.15.7:0/3496119636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b7c60 0x7f30681414a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3058009870 tx=0x7f305801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12a466918437c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:26.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f305ffff640 1 -- 172.21.15.7:0/3496119636 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30580b0070 con 0x7f30680b7c60 2024-03-19T01:24:26.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f305ffff640 1 -- 172.21.15.7:0/3496119636 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30580ab600 con 0x7f30680b7c60 2024-03-19T01:24:26.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f305ffff640 1 -- 172.21.15.7:0/3496119636 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30580ab9d0 con 0x7f30680b7c60 2024-03-19T01:24:26.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306f649640 1 -- 172.21.15.7:0/3496119636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b7c60 msgr2=0x7f30681414a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306f649640 1 --2- 172.21.15.7:0/3496119636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b7c60 0x7f30681414a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3058009870 tx=0x7f305801c520 comp rx=0 tx=0).stop 2024-03-19T01:24:26.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306f649640 1 -- 172.21.15.7:0/3496119636 shutdown_connections 2024-03-19T01:24:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306f649640 1 --2- 172.21.15.7:0/3496119636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b7c60 0x7f30681414a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306f649640 1 --2- 172.21.15.7:0/3496119636 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b6380 0x7f30680b7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306f649640 1 -- 172.21.15.7:0/3496119636 >> 172.21.15.7:0/3496119636 conn(0x7f306806b2c0 msgr2=0x7f306806b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306f649640 1 -- 172.21.15.7:0/3496119636 shutdown_connections 2024-03-19T01:24:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.910+0000 7f306f649640 1 -- 172.21.15.7:0/3496119636 wait complete. 2024-03-19T01:24:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 Processor -- start 2024-03-19T01:24:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 -- start start 2024-03-19T01:24:26.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b6380 0x7f30680ba900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b7c60 0x7f3068144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30681419e0 0x7f3068143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3068155120 con 0x7f30680b6380 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3068154fa0 con 0x7f30681419e0 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f30681552a0 con 0x7f30680b7c60 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306d3e5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b6380 0x7f30680ba900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306d3e5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b6380 0x7f30680ba900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39458/0 (socket says 172.21.15.7:39458) 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306d3e5640 1 -- 172.21.15.7:0/1509520026 learned_addr learned my addr 172.21.15.7:0/1509520026 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306dbe6640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30681419e0 0x7f3068143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306cbe4640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b7c60 0x7f3068144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306d3e5640 1 -- 172.21.15.7:0/1509520026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b7c60 msgr2=0x7f3068144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306d3e5640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b7c60 0x7f3068144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306d3e5640 1 -- 172.21.15.7:0/1509520026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30681419e0 msgr2=0x7f3068143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306d3e5640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30681419e0 0x7f3068143820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306d3e5640 1 -- 172.21.15.7:0/1509520026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3068143120 con 0x7f30680b6380 2024-03-19T01:24:26.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306cbe4640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b7c60 0x7f3068144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:26.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306d3e5640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b6380 0x7f30680ba900 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3064007ec0 tx=0x7f306400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:26.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f305dffb640 1 -- 172.21.15.7:0/1509520026 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064014620 con 0x7f30680b6380 2024-03-19T01:24:26.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f305dffb640 1 -- 172.21.15.7:0/1509520026 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3064019070 con 0x7f30680b6380 2024-03-19T01:24:26.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30680b2fb0 con 0x7f30680b6380 2024-03-19T01:24:26.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f305dffb640 1 -- 172.21.15.7:0/1509520026 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064014a20 con 0x7f30680b6380 2024-03-19T01:24:26.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30680b24d0 con 0x7f30680b6380 2024-03-19T01:24:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3068075570 con 0x7f30680b6380 2024-03-19T01:24:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f305dffb640 1 -- 172.21.15.7:0/1509520026 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3064014bc0 con 0x7f30680b6380 2024-03-19T01:24:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f305dffb640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f304c067690 0x7f304c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.914+0000 7f305dffb640 1 -- 172.21.15.7:0/1509520026 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5795+0+0 (secure 0 0 0) 0x7f30640861e0 con 0x7f30680b6380 2024-03-19T01:24:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.918+0000 7f306cbe4640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f304c067690 0x7f304c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.918+0000 7f306cbe4640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f304c067690 0x7f304c069b50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f30580ad000 tx=0x7f3058008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:26.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:26.922+0000 7f305dffb640 1 -- 172.21.15.7:0/1509520026 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3064056e90 con 0x7f30680b6380 2024-03-19T01:24:27.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.086+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3068143390 con 0x7f30680b6380 2024-03-19T01:24:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.994+0000 7f305dffb640 1 -- 172.21.15.7:0/1509520026 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v32) ==== 181+0+0 (secure 0 0 0) 0x7f3064056790 con 0x7f30680b6380 2024-03-19T01:24:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' removed 2024-03-19T01:24:28.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f304c067690 msgr2=0x7f304c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:28.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f304c067690 0x7f304c069b50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f30580ad000 tx=0x7f3058008040 comp rx=0 tx=0).stop 2024-03-19T01:24:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b6380 msgr2=0x7f30680ba900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b6380 0x7f30680ba900 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3064007ec0 tx=0x7f306400e4b0 comp rx=0 tx=0).stop 2024-03-19T01:24:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 shutdown_connections 2024-03-19T01:24:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f304c067690 0x7f304c069b50 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f30580ad000 tx=0x7f3058008040 comp rx=0 tx=0).stop 2024-03-19T01:24:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30680b7c60 0x7f3068144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30681419e0 0x7f3068143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 --2- 172.21.15.7:0/1509520026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30680b6380 0x7f30680ba900 secure :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3064007ec0 tx=0x7f306400e4b0 comp rx=0 tx=0).stop 2024-03-19T01:24:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 >> 172.21.15.7:0/1509520026 conn(0x7f306806b2c0 msgr2=0x7f3068067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:27.998+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 shutdown_connections 2024-03-19T01:24:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.002+0000 7f306f649640 1 -- 172.21.15.7:0/1509520026 wait complete. 2024-03-19T01:24:28.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-19T01:24:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.150+0000 7f8b04b05640 1 Processor -- start 2024-03-19T01:24:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.150+0000 7f8b04b05640 1 -- start start 2024-03-19T01:24:28.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8b04b05640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b000b6140 0x7f8b00141430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8b04b05640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b000b6520 con 0x7f8b00141970 2024-03-19T01:24:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8b04b05640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b0006d350 con 0x7f8b000b6f50 2024-03-19T01:24:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8b04b05640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8b000b7530 con 0x7f8b000b6140 2024-03-19T01:24:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afdd9b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8b00141970 0x7f8b000b6840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35514/0 (socket says 172.21.15.7:35514) 2024-03-19T01:24:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afdd9b640 1 -- 172.21.15.7:0/2856316265 learned_addr learned my addr 172.21.15.7:0/2856316265 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afe59c640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b000b6140 0x7f8b00141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895101227 0 0) 0x7f8b000b6520 con 0x7f8b00141970 2024-03-19T01:24:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ae8003530 con 0x7f8b00141970 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 287431005 0 0) 0x7f8ae8003530 con 0x7f8b00141970 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b000b6140 msgr2=0x7f8b00141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b000b6140 0x7f8b00141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 >> v1:172.21.15.167:6789/0 conn(0x7f8b000b6f50 legacy=0x7f8b00151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b000b7770 con 0x7f8b00141970 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f8af4001680 con 0x7f8b00141970 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 >> v1:172.21.15.7:6789/0 conn(0x7f8b00141970 legacy=0x7f8b000b6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8ae8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b000b6140 0x7f8ae8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ae8004d50 0x7f8ae80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8b000b7530 con 0x7f8ae80039f0 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae800bab0 con 0x7f8ae8004d50 2024-03-19T01:24:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8ae800bcb0 con 0x7f8b000b6140 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afe59c640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8ae8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afed9d640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ae8004d50 0x7f8ae80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afdd9b640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b000b6140 0x7f8ae8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afe59c640 1 -- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b000b6140 msgr2=0x7f8ae8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afe59c640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b000b6140 0x7f8ae8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afe59c640 1 -- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ae8004d50 msgr2=0x7f8ae80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afe59c640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ae8004d50 0x7f8ae80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afe59c640 1 -- 172.21.15.7:0/2856316265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae8003530 con 0x7f8ae80039f0 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afed9d640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ae8004d50 0x7f8ae80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afdd9b640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b000b6140 0x7f8ae8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.154+0000 7f8afe59c640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8ae8003e10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009ab0 tx=0x7f8af001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78a4ed9b456d06a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af00b2070 con 0x7f8ae80039f0 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af00a5530 con 0x7f8ae80039f0 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afd59a640 1 -- 172.21.15.7:0/2856316265 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af00096b0 con 0x7f8ae80039f0 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- 172.21.15.7:0/2856316265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 msgr2=0x7f8ae8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 --2- 172.21.15.7:0/2856316265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8ae8003e10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009ab0 tx=0x7f8af001ca50 comp rx=0 tx=0).stop 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afed9d640 1 -- 172.21.15.7:0/2856316265 reap_dead start 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- 172.21.15.7:0/2856316265 shutdown_connections 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- 172.21.15.7:0/2856316265 >> 172.21.15.7:0/2856316265 conn(0x7f8b0006b2c0 msgr2=0x7f8b0006b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- 172.21.15.7:0/2856316265 shutdown_connections 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- 172.21.15.7:0/2856316265 wait complete. 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 Processor -- start 2024-03-19T01:24:28.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- start start 2024-03-19T01:24:28.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8b00147d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:28.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b001e05a0 0x7f8b001e43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b00139ce0 0x7f8b000b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8b00153ee0 con 0x7f8ae80039f0 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8b00072c90 con 0x7f8b00139ce0 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8b00154060 con 0x7f8b001e05a0 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afe59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8b00147d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afe59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8b00147d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39470/0 (socket says 172.21.15.7:39470) 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afe59c640 1 -- 172.21.15.7:0/847354883 learned_addr learned my addr 172.21.15.7:0/847354883 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afed9d640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b00139ce0 0x7f8b000b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afdd9b640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b001e05a0 0x7f8b001e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afe59c640 1 -- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b001e05a0 msgr2=0x7f8b001e43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afe59c640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b001e05a0 0x7f8b001e43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afe59c640 1 -- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b00139ce0 msgr2=0x7f8b000b8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afe59c640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b00139ce0 0x7f8b000b8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:28.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afe59c640 1 -- 172.21.15.7:0/847354883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b001437e0 con 0x7f8ae80039f0 2024-03-19T01:24:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afed9d640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b00139ce0 0x7f8b000b8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afdd9b640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b001e05a0 0x7f8b001e43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8afe59c640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8b00147d10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8af001c880 tx=0x7f8ae8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8aeeffd640 1 -- 172.21.15.7:0/847354883 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af00bc020 con 0x7f8ae80039f0 2024-03-19T01:24:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8aeeffd640 1 -- 172.21.15.7:0/847354883 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af00a5d60 con 0x7f8ae80039f0 2024-03-19T01:24:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b000b9ee0 con 0x7f8ae80039f0 2024-03-19T01:24:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8aeeffd640 1 -- 172.21.15.7:0/847354883 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af00b2040 con 0x7f8ae80039f0 2024-03-19T01:24:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.158+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b001e3bb0 con 0x7f8ae80039f0 2024-03-19T01:24:28.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.162+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b00075570 con 0x7f8ae80039f0 2024-03-19T01:24:28.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.162+0000 7f8aeeffd640 1 -- 172.21.15.7:0/847354883 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8af0004220 con 0x7f8ae80039f0 2024-03-19T01:24:28.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.162+0000 7f8aeeffd640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8ae0067690 0x7f8ae0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:28.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.162+0000 7f8aeeffd640 1 -- 172.21.15.7:0/847354883 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5357+0+0 (secure 0 0 0) 0x7f8af011f6e0 con 0x7f8ae80039f0 2024-03-19T01:24:28.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.162+0000 7f8afdd9b640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8ae0067690 0x7f8ae0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:28.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.162+0000 7f8afdd9b640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8ae0067690 0x7f8ae0069b50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b0013d9b0 tx=0x7f8af4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.166+0000 7f8aeeffd640 1 -- 172.21.15.7:0/847354883 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8af00c6020 con 0x7f8ae80039f0 2024-03-19T01:24:28.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:28.326+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8b001422b0 con 0x7f8ae80039f0 2024-03-19T01:24:29.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.002+0000 7f8aeeffd640 1 -- 172.21.15.7:0/847354883 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v33) ==== 169+0+0 (secure 0 0 0) 0x7f8af0016760 con 0x7f8ae80039f0 2024-03-19T01:24:29.004 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' removed 2024-03-19T01:24:29.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8ae0067690 msgr2=0x7f8ae0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8ae0067690 0x7f8ae0069b50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b0013d9b0 tx=0x7f8af4009040 comp rx=0 tx=0).stop 2024-03-19T01:24:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 msgr2=0x7f8b00147d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8b00147d10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8af001c880 tx=0x7f8ae8005840 comp rx=0 tx=0).stop 2024-03-19T01:24:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 shutdown_connections 2024-03-19T01:24:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8ae0067690 0x7f8ae0069b50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b001e05a0 0x7f8b001e43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b00139ce0 0x7f8b000b8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 --2- 172.21.15.7:0/847354883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae80039f0 0x7f8b00147d10 secure :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8af001c880 tx=0x7f8ae8005840 comp rx=0 tx=0).stop 2024-03-19T01:24:29.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.006+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 >> 172.21.15.7:0/847354883 conn(0x7f8b0006b2c0 msgr2=0x7f8b0013d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:29.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.010+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 shutdown_connections 2024-03-19T01:24:29.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.010+0000 7f8b04b05640 1 -- 172.21.15.7:0/847354883 wait complete. 2024-03-19T01:24:29.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-19T01:24:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.162+0000 7fb895e25640 1 Processor -- start 2024-03-19T01:24:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.162+0000 7fb895e25640 1 -- start start 2024-03-19T01:24:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb895e25640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fb890144bd0 0x7fb89006bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb895e25640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb890072730 con 0x7fb8900b1430 2024-03-19T01:24:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb895e25640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb890072a10 con 0x7fb8901447f0 2024-03-19T01:24:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb895e25640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fb8900affa0 con 0x7fb890144bd0 2024-03-19T01:24:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb894e23640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fb8901447f0 0x7fb8900b0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:55358/0 (socket says 172.21.15.7:55358) 2024-03-19T01:24:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb894e23640 1 -- 172.21.15.7:0/1432206231 learned_addr learned my addr 172.21.15.7:0/1432206231 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb895624640 1 --2- 172.21.15.7:0/1432206231 >> v2:172.21.15.167:3300/0 conn(0x7fb890144bd0 0x7fb89006bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb88f7fe640 1 -- 172.21.15.7:0/1432206231 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923467180 0 0) 0x7fb890072730 con 0x7fb8900b1430 2024-03-19T01:24:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb88f7fe640 1 -- 172.21.15.7:0/1432206231 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb878003530 con 0x7fb8900b1430 2024-03-19T01:24:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb88f7fe640 1 -- 172.21.15.7:0/1432206231 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3033555901 0 0) 0x7fb890072a10 con 0x7fb8901447f0 2024-03-19T01:24:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb895624640 1 -- 172.21.15.7:0/1432206231 >> v1:172.21.15.167:6789/0 conn(0x7fb8901447f0 legacy=0x7fb8900b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb895624640 1 -- 172.21.15.7:0/1432206231 >> v1:172.21.15.7:6789/0 conn(0x7fb8900b1430 legacy=0x7fb89006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.166+0000 7fb895624640 1 -- 172.21.15.7:0/1432206231 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8900b1d00 con 0x7fb890144bd0 2024-03-19T01:24:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.170+0000 7fb895624640 1 --2- 172.21.15.7:0/1432206231 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890144bd0 0x7fb89006bda0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb8880097c0 tx=0x7fb88801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=427278d3c8ec7264 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.170+0000 7fb88f7fe640 1 -- 172.21.15.7:0/1432206231 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880b0070 con 0x7fb890144bd0 2024-03-19T01:24:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb88f7fe640 1 -- 172.21.15.7:0/1432206231 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8880ab600 con 0x7fb890144bd0 2024-03-19T01:24:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb88f7fe640 1 -- 172.21.15.7:0/1432206231 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880ab9d0 con 0x7fb890144bd0 2024-03-19T01:24:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb895e25640 1 -- 172.21.15.7:0/1432206231 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890144bd0 msgr2=0x7fb89006bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb895e25640 1 --2- 172.21.15.7:0/1432206231 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890144bd0 0x7fb89006bda0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb8880097c0 tx=0x7fb88801c380 comp rx=0 tx=0).stop 2024-03-19T01:24:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb895e25640 1 -- 172.21.15.7:0/1432206231 shutdown_connections 2024-03-19T01:24:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb895e25640 1 --2- 172.21.15.7:0/1432206231 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890144bd0 0x7fb89006bda0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb895e25640 1 -- 172.21.15.7:0/1432206231 >> 172.21.15.7:0/1432206231 conn(0x7fb8900b5160 msgr2=0x7fb8900b5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb895e25640 1 -- 172.21.15.7:0/1432206231 shutdown_connections 2024-03-19T01:24:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb895e25640 1 -- 172.21.15.7:0/1432206231 wait complete. 2024-03-19T01:24:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.174+0000 7fb895e25640 1 Processor -- start 2024-03-19T01:24:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb895e25640 1 -- start start 2024-03-19T01:24:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb895e25640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8901447f0 0x7fb890140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb895e25640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890144bd0 0x7fb89013e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb895e25640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb890155140 0x7fb89013ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb895e25640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb89014db60 con 0x7fb8901447f0 2024-03-19T01:24:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb895e25640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb89014d9e0 con 0x7fb890144bd0 2024-03-19T01:24:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb895e25640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb89014dce0 con 0x7fb890155140 2024-03-19T01:24:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb894e23640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8901447f0 0x7fb890140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb894e23640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8901447f0 0x7fb890140da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39474/0 (socket says 172.21.15.7:39474) 2024-03-19T01:24:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb894e23640 1 -- 172.21.15.7:0/2246299422 learned_addr learned my addr 172.21.15.7:0/2246299422 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb88ffff640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890144bd0 0x7fb89013e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb894e23640 1 -- 172.21.15.7:0/2246299422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb890155140 msgr2=0x7fb89013ed80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:24:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb894e23640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb890155140 0x7fb89013ed80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb894e23640 1 -- 172.21.15.7:0/2246299422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890144bd0 msgr2=0x7fb89013e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb894e23640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890144bd0 0x7fb89013e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb894e23640 1 -- 172.21.15.7:0/2246299422 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb890155690 con 0x7fb8901447f0 2024-03-19T01:24:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb894e23640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8901447f0 0x7fb890140da0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb88000c4d0 tx=0x7fb88000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb88d7fa640 1 -- 172.21.15.7:0/2246299422 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880018070 con 0x7fb8901447f0 2024-03-19T01:24:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb890153e80 con 0x7fb8901447f0 2024-03-19T01:24:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8900af6d0 con 0x7fb8901447f0 2024-03-19T01:24:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb88d7fa640 1 -- 172.21.15.7:0/2246299422 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb880007500 con 0x7fb8901447f0 2024-03-19T01:24:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.178+0000 7fb88d7fa640 1 -- 172.21.15.7:0/2246299422 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880004a30 con 0x7fb8901447f0 2024-03-19T01:24:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.182+0000 7fb88d7fa640 1 -- 172.21.15.7:0/2246299422 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb88001b070 con 0x7fb8901447f0 2024-03-19T01:24:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.182+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8700052a0 con 0x7fb8901447f0 2024-03-19T01:24:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.182+0000 7fb88d7fa640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb87c067660 0x7fb87c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.182+0000 7fb88d7fa640 1 -- 172.21.15.7:0/2246299422 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fb880084940 con 0x7fb8901447f0 2024-03-19T01:24:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.182+0000 7fb88ffff640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb87c067660 0x7fb87c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.182+0000 7fb88ffff640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb87c067660 0x7fb87c069b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb884007fd0 tx=0x7fb884009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:29.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.186+0000 7fb88d7fa640 1 -- 172.21.15.7:0/2246299422 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb880055a10 con 0x7fb8901447f0 2024-03-19T01:24:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.350+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fb870005da0 con 0x7fb8901447f0 2024-03-19T01:24:29.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb88d7fa640 1 -- 172.21.15.7:0/2246299422 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fb880055310 con 0x7fb8901447f0 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb87c067660 msgr2=0x7fb87c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb87c067660 0x7fb87c069b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb884007fd0 tx=0x7fb884009040 comp rx=0 tx=0).stop 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8901447f0 msgr2=0x7fb890140da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8901447f0 0x7fb890140da0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb88000c4d0 tx=0x7fb88000c990 comp rx=0 tx=0).stop 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 shutdown_connections 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb87c067660 0x7fb87c069b20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb890155140 0x7fb89013ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb890144bd0 0x7fb89013e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 --2- 172.21.15.7:0/2246299422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8901447f0 0x7fb890140da0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 >> 172.21.15.7:0/2246299422 conn(0x7fb8900b5160 msgr2=0x7fb8900b0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 shutdown_connections 2024-03-19T01:24:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.354+0000 7fb895e25640 1 -- 172.21.15.7:0/2246299422 wait complete. 2024-03-19T01:24:29.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args' 2024-03-19T01:24:29.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b20a5c640 1 Processor -- start 2024-03-19T01:24:29.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b20a5c640 1 -- start start 2024-03-19T01:24:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b20a5c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2b1c0b81c0 0x7f2b1c1486e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b20a5c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2b1c06f640 0x7f2b1c14fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b20a5c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b1c0b85a0 con 0x7f2b1c06f1c0 2024-03-19T01:24:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b20a5c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f2b1c0b2710 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b20a5c640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f2b1c070960 con 0x7f2b1c06f640 2024-03-19T01:24:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b1a59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2b1c0b81c0 0x7f2b1c1486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b1a59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2b1c0b81c0 0x7f2b1c1486e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39486/0 (socket says 172.21.15.7:39486) 2024-03-19T01:24:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b1a59c640 1 -- 172.21.15.7:0/3627056000 learned_addr learned my addr 172.21.15.7:0/3627056000 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b1ad9d640 1 --2- 172.21.15.7:0/3627056000 >> v2:172.21.15.167:3300/0 conn(0x7f2b1c06f640 0x7f2b1c14fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b1a59c640 1 -- 172.21.15.7:0/3627056000 >> v2:172.21.15.167:3300/0 conn(0x7f2b1c06f640 msgr2=0x7f2b1c14fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b1a59c640 1 --2- 172.21.15.7:0/3627056000 >> v2:172.21.15.167:3300/0 conn(0x7f2b1c06f640 0x7f2b1c14fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b1a59c640 1 -- 172.21.15.7:0/3627056000 >> v1:172.21.15.167:6789/0 conn(0x7f2b1c06f1c0 legacy=0x7f2b1c13f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.494+0000 7f2b1a59c640 1 -- 172.21.15.7:0/3627056000 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c0b20c0 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b1a59c640 1 --2- 172.21.15.7:0/3627056000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 0x7f2b1c1486e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c06b790 tx=0x7f2b0c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab66add216a371b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:29.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b19d9b640 1 -- 172.21.15.7:0/3627056000 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c0ab640 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b19d9b640 1 -- 172.21.15.7:0/3627056000 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b0c0b0070 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b19d9b640 1 -- 172.21.15.7:0/3627056000 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c0aba00 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 -- 172.21.15.7:0/3627056000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 msgr2=0x7f2b1c1486e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 --2- 172.21.15.7:0/3627056000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 0x7f2b1c1486e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c06b790 tx=0x7f2b0c01c420 comp rx=0 tx=0).stop 2024-03-19T01:24:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 -- 172.21.15.7:0/3627056000 shutdown_connections 2024-03-19T01:24:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 --2- 172.21.15.7:0/3627056000 >> v2:172.21.15.167:3300/0 conn(0x7f2b1c06f640 0x7f2b1c14fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 --2- 172.21.15.7:0/3627056000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 0x7f2b1c1486e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 -- 172.21.15.7:0/3627056000 >> 172.21.15.7:0/3627056000 conn(0x7f2b1c0b5330 msgr2=0x7f2b1c0b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:29.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 -- 172.21.15.7:0/3627056000 shutdown_connections 2024-03-19T01:24:29.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 -- 172.21.15.7:0/3627056000 wait complete. 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 Processor -- start 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 -- start start 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b1c06f1c0 0x7f2b1c13d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b1c06f640 0x7f2b1c1e10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 0x7f2b1c1e1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c0b2d20 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c0b2ba0 con 0x7f2b1c06f640 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b20a5c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1c1527d0 con 0x7f2b1c06f1c0 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b1ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 0x7f2b1c1e1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b1ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 0x7f2b1c1e1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39490/0 (socket says 172.21.15.7:39490) 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.498+0000 7f2b1ad9d640 1 -- 172.21.15.7:0/4174523089 learned_addr learned my addr 172.21.15.7:0/4174523089 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b13fff640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b1c06f640 0x7f2b1c1e10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b1a59c640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b1c06f1c0 0x7f2b1c13d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b1ad9d640 1 -- 172.21.15.7:0/4174523089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b1c06f1c0 msgr2=0x7f2b1c13d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b1ad9d640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b1c06f1c0 0x7f2b1c13d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b1ad9d640 1 -- 172.21.15.7:0/4174523089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b1c06f640 msgr2=0x7f2b1c1e10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b1ad9d640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b1c06f640 0x7f2b1c1e10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b1ad9d640 1 -- 172.21.15.7:0/4174523089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c145d60 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b1ad9d640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 0x7f2b1c1e1610 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2b14007e60 tx=0x7f2b1400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b12ffd640 1 -- 172.21.15.7:0/4174523089 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b14014620 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b12ffd640 1 -- 172.21.15.7:0/4174523089 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b14019070 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b12ffd640 1 -- 172.21.15.7:0/4174523089 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b14014a00 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1c1e0bf0 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b1c1408b0 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b1c1e0dc0 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b12ffd640 1 -- 172.21.15.7:0/4174523089 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2b14014ba0 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b12ffd640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2afc067660 0x7f2afc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:29.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b12ffd640 1 -- 172.21.15.7:0/4174523089 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f2b14085df0 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b1a59c640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2afc067660 0x7f2afc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:29.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.502+0000 7f2b1a59c640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2afc067660 0x7f2afc069b20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c0af000 tx=0x7f2b0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:29.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.506+0000 7f2b12ffd640 1 -- 172.21.15.7:0/4174523089 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2b14056e00 con 0x7f2b1c0b81c0 2024-03-19T01:24:29.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:29.722+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]} v 0) -- 0x7f2b1c075460 con 0x7f2b1c0b81c0 2024-03-19T01:24:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.002+0000 7f2b12ffd640 1 -- 172.21.15.7:0/4174523089 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]}]=0 v85) ==== 134+0+0 (secure 0 0 0) 0x7f2b14056700 con 0x7f2b1c0b81c0 2024-03-19T01:24:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2afc067660 msgr2=0x7f2afc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2afc067660 0x7f2afc069b20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c0af000 tx=0x7f2b0c008040 comp rx=0 tx=0).stop 2024-03-19T01:24:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 msgr2=0x7f2b1c1e1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 0x7f2b1c1e1610 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2b14007e60 tx=0x7f2b1400e4b0 comp rx=0 tx=0).stop 2024-03-19T01:24:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 shutdown_connections 2024-03-19T01:24:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2afc067660 0x7f2afc069b20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c0b81c0 0x7f2b1c1e1610 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b1c06f640 0x7f2b1c1e10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 --2- 172.21.15.7:0/4174523089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b1c06f1c0 0x7f2b1c13d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 >> 172.21.15.7:0/4174523089 conn(0x7f2b1c0b5330 msgr2=0x7f2b1c0b6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 shutdown_connections 2024-03-19T01:24:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:30.006+0000 7f2b20a5c640 1 -- 172.21.15.7:0/4174523089 wait complete. 2024-03-19T01:24:30.030 INFO:tasks.cephfs_test_runner:Test that when cd is issued with an argument, CWD is changed ... ERROR 2024-03-19T01:24:30.030 INFO:tasks.cephfs_test_runner: 2024-03-19T01:24:30.030 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-19T01:24:30.030 INFO:tasks.cephfs_test_runner:ERROR: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-19T01:24:30.030 INFO:tasks.cephfs_test_runner:Test that when cd is issued with an argument, CWD is changed 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/test_cephfs_shell.py", line 602, in test_cd_with_args 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: output = self.get_cephfs_shell_script_output(script) 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/test_cephfs_shell.py", line 161, in get_cephfs_shell_script_output 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: return ensure_str(self.run_cephfs_shell_script( 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/test_cephfs_shell.py", line 153, in run_cephfs_shell_script 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: return mount_x.client_remote.run(args=args, stdout=stdout, 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-19T01:24:30.031 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 127: 'cephfs-shell -b /tmp/test-cephfsmp5s52lw' 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner: 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner:Ran 1 test in 39.731s 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner: 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner: 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner:ERROR: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner:Test that when cd is issued with an argument, CWD is changed 2024-03-19T01:24:30.032 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/test_cephfs_shell.py", line 602, in test_cd_with_args 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: output = self.get_cephfs_shell_script_output(script) 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/test_cephfs_shell.py", line 161, in get_cephfs_shell_script_output 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: return ensure_str(self.run_cephfs_shell_script( 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/test_cephfs_shell.py", line 153, in run_cephfs_shell_script 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: return mount_x.client_remote.run(args=args, stdout=stdout, 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 127: 'cephfs-shell -b /tmp/test-cephfsmp5s52lw' 2024-03-19T01:24:30.033 INFO:tasks.cephfs_test_runner: 2024-03-19T01:24:30.034 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-19T01:24:30.863 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c0f678e149b7408f8988b1edd9e9aad3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-19T01:24:30.865 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-19T01:24:30.874 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-19T01:24:30.897 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-19T01:24:30.897 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-19T01:24:30.898 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:24:30.912 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16339616k,nr_inodes=4084904,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17778 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275312k,nr_inodes=818828,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-19T01:24:30.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:30.913 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-03-19T01:24:30.963 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-19T01:24:30.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:30.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T01:24:31.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:24:31.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2024-03-19T01:24:31.210 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-19T01:24:31.229 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-19T01:24:31.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:24:31.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.366+0000 7faddda3a640 1 Processor -- start 2024-03-19T01:24:31.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.366+0000 7faddda3a640 1 -- start start 2024-03-19T01:24:31.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7faddda3a640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fadd80b1460 0x7fadd80b6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7faddda3a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd806d870 0x7fadd8144700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7faddda3a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadd80b80f0 0x7fadd8144c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7faddda3a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fadd80b1840 con 0x7fadd80b80f0 2024-03-19T01:24:31.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7faddda3a640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fadd80b2120 con 0x7fadd80b1460 2024-03-19T01:24:31.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7faddda3a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fadd81441e0 con 0x7fadd806d870 2024-03-19T01:24:31.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadd77fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadd80b80f0 0x7fadd8144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadd77fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadd80b80f0 0x7fadd8144c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39498/0 (socket says 172.21.15.7:39498) 2024-03-19T01:24:31.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadd77fe640 1 -- 172.21.15.7:0/2062870313 learned_addr learned my addr 172.21.15.7:0/2062870313 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:31.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadcffff640 1 --2- 172.21.15.7:0/2062870313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd806d870 0x7fadd8144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadd6ffd640 1 --2- 172.21.15.7:0/2062870313 >> v2:172.21.15.167:3300/0 conn(0x7fadd80b1460 0x7fadd80b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadd77fe640 1 -- 172.21.15.7:0/2062870313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd806d870 msgr2=0x7fadd8144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:31.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadd77fe640 1 --2- 172.21.15.7:0/2062870313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd806d870 0x7fadd8144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadd77fe640 1 -- 172.21.15.7:0/2062870313 >> v2:172.21.15.167:3300/0 conn(0x7fadd80b1460 msgr2=0x7fadd80b6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:31.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadd77fe640 1 --2- 172.21.15.7:0/2062870313 >> v2:172.21.15.167:3300/0 conn(0x7fadd80b1460 0x7fadd80b6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.370+0000 7fadd77fe640 1 -- 172.21.15.7:0/2062870313 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd80b85c0 con 0x7fadd80b80f0 2024-03-19T01:24:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 --2- 172.21.15.7:0/2062870313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 0x7fadd8144c40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fadd80b9920 tx=0x7fadd001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4b2744b6d209ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd67fc640 1 -- 172.21.15.7:0/2062870313 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd00a55e0 con 0x7fadd80b80f0 2024-03-19T01:24:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd67fc640 1 -- 172.21.15.7:0/2062870313 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadd00a5780 con 0x7fadd80b80f0 2024-03-19T01:24:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd67fc640 1 -- 172.21.15.7:0/2062870313 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd00a5b50 con 0x7fadd80b80f0 2024-03-19T01:24:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- 172.21.15.7:0/2062870313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 msgr2=0x7fadd8144c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 --2- 172.21.15.7:0/2062870313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 0x7fadd8144c40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fadd80b9920 tx=0x7fadd001c960 comp rx=0 tx=0).stop 2024-03-19T01:24:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- 172.21.15.7:0/2062870313 shutdown_connections 2024-03-19T01:24:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 --2- 172.21.15.7:0/2062870313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 0x7fadd8144c40 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 --2- 172.21.15.7:0/2062870313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd806d870 0x7fadd8144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 --2- 172.21.15.7:0/2062870313 >> v2:172.21.15.167:3300/0 conn(0x7fadd80b1460 0x7fadd80b6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- 172.21.15.7:0/2062870313 >> 172.21.15.7:0/2062870313 conn(0x7fadd80b5330 msgr2=0x7fadd80b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- 172.21.15.7:0/2062870313 shutdown_connections 2024-03-19T01:24:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- 172.21.15.7:0/2062870313 wait complete. 2024-03-19T01:24:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 Processor -- start 2024-03-19T01:24:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- start start 2024-03-19T01:24:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadd806d870 0x7fadd813a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd80b1460 0x7fadd81ddc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 0x7fadd81e11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fadd81481e0 con 0x7fadd80b80f0 2024-03-19T01:24:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fadd8148060 con 0x7fadd806d870 2024-03-19T01:24:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fadd8148360 con 0x7fadd80b1460 2024-03-19T01:24:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 0x7fadd81e11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 0x7fadd81e11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39512/0 (socket says 172.21.15.7:39512) 2024-03-19T01:24:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 -- 172.21.15.7:0/3699915546 learned_addr learned my addr 172.21.15.7:0/3699915546 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd6ffd640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadd806d870 0x7fadd813a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadcffff640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd80b1460 0x7fadd81ddc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 -- 172.21.15.7:0/3699915546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd80b1460 msgr2=0x7fadd81ddc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd80b1460 0x7fadd81ddc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 -- 172.21.15.7:0/3699915546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadd806d870 msgr2=0x7fadd813a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadd806d870 0x7fadd813a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 -- 172.21.15.7:0/3699915546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd80713d0 con 0x7fadd80b80f0 2024-03-19T01:24:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd77fe640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 0x7fadd81e11c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fadd00af040 tx=0x7fadd0019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd4ff9640 1 -- 172.21.15.7:0/3699915546 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd00bc020 con 0x7fadd80b80f0 2024-03-19T01:24:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd4ff9640 1 -- 172.21.15.7:0/3699915546 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadd00b8670 con 0x7fadd80b80f0 2024-03-19T01:24:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7fadd4ff9640 1 -- 172.21.15.7:0/3699915546 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0009510 con 0x7fadd80b80f0 2024-03-19T01:24:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.374+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd8139d50 con 0x7fadd80b80f0 2024-03-19T01:24:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.378+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd8137330 con 0x7fadd80b80f0 2024-03-19T01:24:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.378+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd8075460 con 0x7fadd80b80f0 2024-03-19T01:24:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.378+0000 7fadd4ff9640 1 -- 172.21.15.7:0/3699915546 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fadd00bf020 con 0x7fadd80b80f0 2024-03-19T01:24:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.378+0000 7fadd4ff9640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fadb8067660 0x7fadb8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.378+0000 7fadd4ff9640 1 -- 172.21.15.7:0/3699915546 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fadd0120890 con 0x7fadd80b80f0 2024-03-19T01:24:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.378+0000 7fadd6ffd640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fadb8067660 0x7fadb8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.378+0000 7fadd6ffd640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fadb8067660 0x7fadb8069b20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fadc800b9d0 tx=0x7fadc8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.382+0000 7fadd4ff9640 1 -- 172.21.15.7:0/3699915546 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fadd00c6030 con 0x7fadd80b80f0 2024-03-19T01:24:31.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.538+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fadd8070380 con 0x7fadb8067660 2024-03-19T01:24:31.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.538+0000 7fadd4ff9640 1 -- 172.21.15.7:0/3699915546 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13482 (secure 0 0 0) 0x7fadd8070380 con 0x7fadb8067660 2024-03-19T01:24:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:31.543 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:24:31.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fadb8067660 msgr2=0x7fadb8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fadb8067660 0x7fadb8069b20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fadc800b9d0 tx=0x7fadc8008040 comp rx=0 tx=0).stop 2024-03-19T01:24:31.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 msgr2=0x7fadd81e11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 0x7fadd81e11c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fadd00af040 tx=0x7fadd0019700 comp rx=0 tx=0).stop 2024-03-19T01:24:31.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 shutdown_connections 2024-03-19T01:24:31.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fadb8067660 0x7fadb8069b20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fadd80b1460 0x7fadd81ddc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80b80f0 0x7fadd81e11c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 --2- 172.21.15.7:0/3699915546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadd806d870 0x7fadd813a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 >> 172.21.15.7:0/3699915546 conn(0x7fadd80b5330 msgr2=0x7fadd813b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:31.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 shutdown_connections 2024-03-19T01:24:31.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.542+0000 7faddda3a640 1 -- 172.21.15.7:0/3699915546 wait complete. 2024-03-19T01:24:31.562 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-19T01:24:30.906965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":3960,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":4055040,"data_stored":2282417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10891,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-54,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002051"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T01:24:26.567867+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:24:26.567867+0000","last_peered":"2024-03-19T01:24:26.567867+0000","last_clean":"2024-03-19T01:24:26.567867+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:24:26.567867+0000","last_undegraded":"2024-03-19T01:24:26.567867+0000","last_fullsized":"2024-03-19T01:24:26.567867+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":725222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31536,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340304,"statfs":{"total":96636764160,"available":96604471296,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":134834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":144879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":725132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:24:31.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:24:31.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab46c6a640 1 Processor -- start 2024-03-19T01:24:31.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab46c6a640 1 -- start start 2024-03-19T01:24:31.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab46c6a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 0x7fab401486e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab46c6a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab4006f640 0x7fab4014fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab46c6a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab400b85a0 con 0x7fab4006f1c0 2024-03-19T01:24:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab46c6a640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fab400b2710 con 0x7fab4006f640 2024-03-19T01:24:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab46c6a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fab40070960 con 0x7fab400b81c0 2024-03-19T01:24:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab45207640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab4006f640 0x7fab4014fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab45207640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab4006f640 0x7fab4014fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39528/0 (socket says 172.21.15.7:39528) 2024-03-19T01:24:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab45207640 1 -- 172.21.15.7:0/1660230322 learned_addr learned my addr 172.21.15.7:0/1660230322 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab44a06640 1 --2- 172.21.15.7:0/1660230322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 0x7fab401486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab45207640 1 -- 172.21.15.7:0/1660230322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 msgr2=0x7fab401486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:31.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab45207640 1 --2- 172.21.15.7:0/1660230322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 0x7fab401486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab45207640 1 -- 172.21.15.7:0/1660230322 >> v1:172.21.15.167:6789/0 conn(0x7fab4006f1c0 legacy=0x7fab4013f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.690+0000 7fab45207640 1 -- 172.21.15.7:0/1660230322 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab400b20c0 con 0x7fab4006f640 2024-03-19T01:24:31.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab45207640 1 --2- 172.21.15.7:0/1660230322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f640 0x7fab4014fa70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fab38009e80 tx=0x7fab3801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585223a18ffb5260 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab3f7fe640 1 -- 172.21.15.7:0/1660230322 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380a5830 con 0x7fab4006f640 2024-03-19T01:24:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab3f7fe640 1 -- 172.21.15.7:0/1660230322 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab380a59d0 con 0x7fab4006f640 2024-03-19T01:24:31.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab3f7fe640 1 -- 172.21.15.7:0/1660230322 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380a5da0 con 0x7fab4006f640 2024-03-19T01:24:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 -- 172.21.15.7:0/1660230322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f640 msgr2=0x7fab4014fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 --2- 172.21.15.7:0/1660230322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f640 0x7fab4014fa70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fab38009e80 tx=0x7fab3801c950 comp rx=0 tx=0).stop 2024-03-19T01:24:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 -- 172.21.15.7:0/1660230322 shutdown_connections 2024-03-19T01:24:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 --2- 172.21.15.7:0/1660230322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f640 0x7fab4014fa70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 --2- 172.21.15.7:0/1660230322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 0x7fab401486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 -- 172.21.15.7:0/1660230322 >> 172.21.15.7:0/1660230322 conn(0x7fab400b5330 msgr2=0x7fab400b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 -- 172.21.15.7:0/1660230322 shutdown_connections 2024-03-19T01:24:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 -- 172.21.15.7:0/1660230322 wait complete. 2024-03-19T01:24:31.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 Processor -- start 2024-03-19T01:24:31.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.694+0000 7fab46c6a640 1 -- start start 2024-03-19T01:24:31.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab46c6a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f1c0 0x7fab4013dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab46c6a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4006f640 0x7fab401e1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab46c6a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 0x7fab40138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab46c6a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fab400b2d00 con 0x7fab4006f1c0 2024-03-19T01:24:31.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab46c6a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fab400b2b80 con 0x7fab4006f640 2024-03-19T01:24:31.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab46c6a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fab401527d0 con 0x7fab400b81c0 2024-03-19T01:24:31.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab44a06640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f1c0 0x7fab4013dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab44a06640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f1c0 0x7fab4013dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39532/0 (socket says 172.21.15.7:39532) 2024-03-19T01:24:31.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab44a06640 1 -- 172.21.15.7:0/1180911088 learned_addr learned my addr 172.21.15.7:0/1180911088 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:31.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab3ffff640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4006f640 0x7fab401e1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab45207640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 0x7fab40138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab44a06640 1 -- 172.21.15.7:0/1180911088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 msgr2=0x7fab40138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab44a06640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 0x7fab40138960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab44a06640 1 -- 172.21.15.7:0/1180911088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4006f640 msgr2=0x7fab401e1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab44a06640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4006f640 0x7fab401e1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab44a06640 1 -- 172.21.15.7:0/1180911088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab401400a0 con 0x7fab4006f1c0 2024-03-19T01:24:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab44a06640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f1c0 0x7fab4013dcb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fab3000d800 tx=0x7fab3000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab3d7fa640 1 -- 172.21.15.7:0/1180911088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3000b6c0 con 0x7fab4006f1c0 2024-03-19T01:24:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab3d7fa640 1 -- 172.21.15.7:0/1180911088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab30019070 con 0x7fab4006f1c0 2024-03-19T01:24:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab3d7fa640 1 -- 172.21.15.7:0/1180911088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3000ba60 con 0x7fab4006f1c0 2024-03-19T01:24:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab40145c30 con 0x7fab4006f1c0 2024-03-19T01:24:31.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab40140f40 con 0x7fab4006f1c0 2024-03-19T01:24:31.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab4013ea40 con 0x7fab4006f1c0 2024-03-19T01:24:31.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab3d7fa640 1 -- 172.21.15.7:0/1180911088 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fab3000bc00 con 0x7fab4006f1c0 2024-03-19T01:24:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab3d7fa640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab3d7fa640 1 -- 172.21.15.7:0/1180911088 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fab30085890 con 0x7fab4006f1c0 2024-03-19T01:24:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab3ffff640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.698+0000 7fab3ffff640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fab40071bd0 tx=0x7fab34009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:31.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.702+0000 7fab3d7fa640 1 -- 172.21.15.7:0/1180911088 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fab30056250 con 0x7fab4006f1c0 2024-03-19T01:24:31.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.862+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab40075460 con 0x7fab2c067660 2024-03-19T01:24:31.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.862+0000 7fab3d7fa640 1 -- 172.21.15.7:0/1180911088 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13482 (secure 0 0 0) 0x7fab40075460 con 0x7fab2c067660 2024-03-19T01:24:31.866 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:31.867 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:24:31.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 msgr2=0x7fab2c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fab40071bd0 tx=0x7fab34009040 comp rx=0 tx=0).stop 2024-03-19T01:24:31.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f1c0 msgr2=0x7fab4013dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:31.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f1c0 0x7fab4013dcb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fab3000d800 tx=0x7fab3000dcc0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 shutdown_connections 2024-03-19T01:24:31.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fab2c067660 0x7fab2c069b20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab400b81c0 0x7fab40138960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab4006f640 0x7fab401e1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 --2- 172.21.15.7:0/1180911088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab4006f1c0 0x7fab4013dcb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:31.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 >> 172.21.15.7:0/1180911088 conn(0x7fab400b5330 msgr2=0x7fab4013ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:31.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 shutdown_connections 2024-03-19T01:24:31.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:31.866+0000 7fab46c6a640 1 -- 172.21.15.7:0/1180911088 wait complete. 2024-03-19T01:24:31.885 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-19T01:24:30.906965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":3960,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":4055040,"data_stored":2282417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10891,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-54,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002051"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T01:24:26.567867+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:24:26.567867+0000","last_peered":"2024-03-19T01:24:26.567867+0000","last_clean":"2024-03-19T01:24:26.567867+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:24:26.567867+0000","last_undegraded":"2024-03-19T01:24:26.567867+0000","last_fullsized":"2024-03-19T01:24:26.567867+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":725222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31536,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340304,"statfs":{"total":96636764160,"available":96604471296,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":134834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":144879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":725132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:24:31.886 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-19T01:24:31.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:24:32.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.006+0000 7f829abc9640 1 Processor -- start 2024-03-19T01:24:32.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.006+0000 7f829abc9640 1 -- start start 2024-03-19T01:24:32.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f829abc9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829406f1c0 0x7f829413f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f829abc9640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f829406f640 0x7f829414fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f829abc9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82940b85a0 con 0x7f82940b81c0 2024-03-19T01:24:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f829abc9640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f82940b2710 con 0x7f829406f640 2024-03-19T01:24:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f829abc9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8294070960 con 0x7f829406f1c0 2024-03-19T01:24:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f8299166640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f829406f640 0x7f829414fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f8299166640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f829406f640 0x7f829414fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39546/0 (socket says 172.21.15.7:39546) 2024-03-19T01:24:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f8299166640 1 -- 172.21.15.7:0/645801572 learned_addr learned my addr 172.21.15.7:0/645801572 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f8293fff640 1 --2- 172.21.15.7:0/645801572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829406f1c0 0x7f829413f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f8299166640 1 -- 172.21.15.7:0/645801572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829406f1c0 msgr2=0x7f829413f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:32.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f8299166640 1 --2- 172.21.15.7:0/645801572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829406f1c0 0x7f829413f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f8299166640 1 -- 172.21.15.7:0/645801572 >> v1:172.21.15.7:6789/0 conn(0x7f82940b81c0 legacy=0x7f82941486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f82937fe640 1 -- 172.21.15.7:0/645801572 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2806909099 0 0) 0x7f82940b85a0 con 0x7f82940b81c0 2024-03-19T01:24:32.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f8299166640 1 -- 172.21.15.7:0/645801572 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82940b20c0 con 0x7f829406f640 2024-03-19T01:24:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.010+0000 7f8299166640 1 --2- 172.21.15.7:0/645801572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 0x7f829414fa70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f82840098a0 tx=0x7f828401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8581c06dfcfc3e49 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f82937fe640 1 -- 172.21.15.7:0/645801572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82840b1070 con 0x7f829406f640 2024-03-19T01:24:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f82937fe640 1 -- 172.21.15.7:0/645801572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82840ac620 con 0x7f829406f640 2024-03-19T01:24:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f82937fe640 1 -- 172.21.15.7:0/645801572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82840ac9f0 con 0x7f829406f640 2024-03-19T01:24:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 -- 172.21.15.7:0/645801572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 msgr2=0x7f829414fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 --2- 172.21.15.7:0/645801572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 0x7f829414fa70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f82840098a0 tx=0x7f828401c4b0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 -- 172.21.15.7:0/645801572 shutdown_connections 2024-03-19T01:24:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 --2- 172.21.15.7:0/645801572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 0x7f829414fa70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 --2- 172.21.15.7:0/645801572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829406f1c0 0x7f829413f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 -- 172.21.15.7:0/645801572 >> 172.21.15.7:0/645801572 conn(0x7f82940b5330 msgr2=0x7f82940b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 -- 172.21.15.7:0/645801572 shutdown_connections 2024-03-19T01:24:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 -- 172.21.15.7:0/645801572 wait complete. 2024-03-19T01:24:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 Processor -- start 2024-03-19T01:24:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 -- start start 2024-03-19T01:24:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f829406f1c0 0x7f82940b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 0x7f82940aec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82940b81c0 0x7f82941e3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8294152b60 con 0x7f829406f640 2024-03-19T01:24:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f82941529e0 con 0x7f829406f1c0 2024-03-19T01:24:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f829abc9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8294152ce0 con 0x7f82940b81c0 2024-03-19T01:24:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8293fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 0x7f82940aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8293fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 0x7f82940aec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39550/0 (socket says 172.21.15.7:39550) 2024-03-19T01:24:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8293fff640 1 -- 172.21.15.7:0/2854146924 learned_addr learned my addr 172.21.15.7:0/2854146924 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8298965640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f829406f1c0 0x7f82940b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8299166640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82940b81c0 0x7f82941e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8293fff640 1 -- 172.21.15.7:0/2854146924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82940b81c0 msgr2=0x7f82941e3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8293fff640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82940b81c0 0x7f82941e3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8293fff640 1 -- 172.21.15.7:0/2854146924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f829406f1c0 msgr2=0x7f82940b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8293fff640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f829406f1c0 0x7f82940b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8293fff640 1 -- 172.21.15.7:0/2854146924 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82940afed0 con 0x7f829406f640 2024-03-19T01:24:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8298965640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f829406f1c0 0x7f82940b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f8293fff640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 0x7f82940aec70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f827c00c7f0 tx=0x7f827c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.014+0000 7f82917fa640 1 -- 172.21.15.7:0/2854146924 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c015070 con 0x7f829406f640 2024-03-19T01:24:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f82917fa640 1 -- 172.21.15.7:0/2854146924 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f827c007820 con 0x7f829406f640 2024-03-19T01:24:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82940b1c60 con 0x7f829406f640 2024-03-19T01:24:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f82917fa640 1 -- 172.21.15.7:0/2854146924 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c007c10 con 0x7f829406f640 2024-03-19T01:24:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82940adcd0 con 0x7f829406f640 2024-03-19T01:24:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82940ae950 con 0x7f829406f640 2024-03-19T01:24:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f82917fa640 1 -- 172.21.15.7:0/2854146924 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f827c018070 con 0x7f829406f640 2024-03-19T01:24:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f82917fa640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8274067660 0x7f8274069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f82917fa640 1 -- 172.21.15.7:0/2854146924 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f827c085570 con 0x7f829406f640 2024-03-19T01:24:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f8298965640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8274067660 0x7f8274069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.018+0000 7f8298965640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8274067660 0x7f8274069b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f828c0012f0 tx=0x7f828c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.022+0000 7f82917fa640 1 -- 172.21.15.7:0/2854146924 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f82940ae950 con 0x7f829406f640 2024-03-19T01:24:32.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.178+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8294075460 con 0x7f8274067660 2024-03-19T01:24:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.178+0000 7f82917fa640 1 -- 172.21.15.7:0/2854146924 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13482 (secure 0 0 0) 0x7f8294075460 con 0x7f8274067660 2024-03-19T01:24:32.183 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:32.184 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:24:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8274067660 msgr2=0x7f8274069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8274067660 0x7f8274069b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f828c0012f0 tx=0x7f828c009040 comp rx=0 tx=0).stop 2024-03-19T01:24:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 msgr2=0x7f82940aec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 0x7f82940aec70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f827c00c7f0 tx=0x7f827c00ccb0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 shutdown_connections 2024-03-19T01:24:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f8274067660 0x7f8274069b20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82940b81c0 0x7f82941e3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f829406f640 0x7f82940aec70 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 --2- 172.21.15.7:0/2854146924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f829406f1c0 0x7f82940b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 >> 172.21.15.7:0/2854146924 conn(0x7f82940b5330 msgr2=0x7f82940b6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 shutdown_connections 2024-03-19T01:24:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.182+0000 7f829abc9640 1 -- 172.21.15.7:0/2854146924 wait complete. 2024-03-19T01:24:32.202 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-19T01:24:30.906965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":3960,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":4055040,"data_stored":2282417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10891,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-54,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002051"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T01:24:26.567867+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:24:26.567867+0000","last_peered":"2024-03-19T01:24:26.567867+0000","last_clean":"2024-03-19T01:24:26.567867+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:24:26.567867+0000","last_undegraded":"2024-03-19T01:24:26.567867+0000","last_fullsized":"2024-03-19T01:24:26.567867+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":725222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31536,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340304,"statfs":{"total":96636764160,"available":96604471296,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":134834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":372736,"data_stored":144879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":725132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:24:32.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:24:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.314+0000 7f077cdb1640 1 Processor -- start 2024-03-19T01:24:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.314+0000 7f077cdb1640 1 -- start start 2024-03-19T01:24:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f077814b0c0 0x7f0778152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07780ba2b0 con 0x7f07780b4790 2024-03-19T01:24:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07780b52b0 con 0x7f07780b9ed0 2024-03-19T01:24:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f07780b6cc0 con 0x7f077814b0c0 2024-03-19T01:24:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f0775d9b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f07780b4790 0x7f0778141010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35544/0 (socket says 172.21.15.7:35544) 2024-03-19T01:24:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f0775d9b640 1 -- 172.21.15.7:0/3918665971 learned_addr learned my addr 172.21.15.7:0/3918665971 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f0776d9d640 1 --2- 172.21.15.7:0/3918665971 >> v2:172.21.15.7:3300/0 conn(0x7f077814b0c0 0x7f0778152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077559a640 1 -- 172.21.15.7:0/3918665971 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3206475361 0 0) 0x7f07780ba2b0 con 0x7f07780b4790 2024-03-19T01:24:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077559a640 1 -- 172.21.15.7:0/3918665971 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0760003530 con 0x7f07780b4790 2024-03-19T01:24:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f0776d9d640 1 -- 172.21.15.7:0/3918665971 >> v1:172.21.15.167:6789/0 conn(0x7f07780b9ed0 legacy=0x7f077814a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f0776d9d640 1 -- 172.21.15.7:0/3918665971 >> v1:172.21.15.7:6789/0 conn(0x7f07780b4790 legacy=0x7f0778141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f0776d9d640 1 -- 172.21.15.7:0/3918665971 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07780b5590 con 0x7f077814b0c0 2024-03-19T01:24:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f0776d9d640 1 --2- 172.21.15.7:0/3918665971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077814b0c0 0x7f0778152cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0770009d90 tx=0x7f077001c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad1af6fcd63c602 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077559a640 1 -- 172.21.15.7:0/3918665971 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07700b2070 con 0x7f077814b0c0 2024-03-19T01:24:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077559a640 1 -- 172.21.15.7:0/3918665971 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07700a58c0 con 0x7f077814b0c0 2024-03-19T01:24:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077559a640 1 -- 172.21.15.7:0/3918665971 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07700a5c90 con 0x7f077814b0c0 2024-03-19T01:24:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 -- 172.21.15.7:0/3918665971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077814b0c0 msgr2=0x7f0778152cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 --2- 172.21.15.7:0/3918665971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077814b0c0 0x7f0778152cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0770009d90 tx=0x7f077001c7e0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 -- 172.21.15.7:0/3918665971 shutdown_connections 2024-03-19T01:24:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 --2- 172.21.15.7:0/3918665971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077814b0c0 0x7f0778152cc0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 -- 172.21.15.7:0/3918665971 >> 172.21.15.7:0/3918665971 conn(0x7f0778072df0 msgr2=0x7f0778073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 -- 172.21.15.7:0/3918665971 shutdown_connections 2024-03-19T01:24:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.318+0000 7f077cdb1640 1 -- 172.21.15.7:0/3918665971 wait complete. 2024-03-19T01:24:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 Processor -- start 2024-03-19T01:24:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 -- start start 2024-03-19T01:24:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07780b4790 0x7f07781e3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07780b9ed0 0x7f07781e3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f077814b0c0 0x7f0778138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0778155ba0 con 0x7f07780b9ed0 2024-03-19T01:24:32.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0778155a20 con 0x7f07780b4790 2024-03-19T01:24:32.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0778155d20 con 0x7f077814b0c0 2024-03-19T01:24:32.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0775d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07780b9ed0 0x7f07781e3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0775d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07780b9ed0 0x7f07781e3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39556/0 (socket says 172.21.15.7:39556) 2024-03-19T01:24:32.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0775d9b640 1 -- 172.21.15.7:0/2846171250 learned_addr learned my addr 172.21.15.7:0/2846171250 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:32.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077659c640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07780b4790 0x7f07781e3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0776d9d640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f077814b0c0 0x7f0778138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0775d9b640 1 -- 172.21.15.7:0/2846171250 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f077814b0c0 msgr2=0x7f0778138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0775d9b640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f077814b0c0 0x7f0778138a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0775d9b640 1 -- 172.21.15.7:0/2846171250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07780b4790 msgr2=0x7f07781e3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0775d9b640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07780b4790 0x7f07781e3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0775d9b640 1 -- 172.21.15.7:0/2846171250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077813d090 con 0x7f07780b9ed0 2024-03-19T01:24:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077659c640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07780b4790 0x7f07781e3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0775d9b640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07780b9ed0 0x7f07781e3980 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f076800c390 tx=0x7f076800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0766ffd640 1 -- 172.21.15.7:0/2846171250 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768017070 con 0x7f07780b9ed0 2024-03-19T01:24:32.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0766ffd640 1 -- 172.21.15.7:0/2846171250 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0768005d10 con 0x7f07780b9ed0 2024-03-19T01:24:32.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f0766ffd640 1 -- 172.21.15.7:0/2846171250 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768007310 con 0x7f07780b9ed0 2024-03-19T01:24:32.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f077813cdb0 con 0x7f07780b9ed0 2024-03-19T01:24:32.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f077813b730 con 0x7f07780b9ed0 2024-03-19T01:24:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.322+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0778075460 con 0x7f07780b9ed0 2024-03-19T01:24:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.326+0000 7f0766ffd640 1 -- 172.21.15.7:0/2846171250 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f076801a070 con 0x7f07780b9ed0 2024-03-19T01:24:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.326+0000 7f0766ffd640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0758067660 0x7f0758069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.326+0000 7f0766ffd640 1 -- 172.21.15.7:0/2846171250 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f0768084950 con 0x7f07780b9ed0 2024-03-19T01:24:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.326+0000 7f077659c640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0758067660 0x7f0758069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.326+0000 7f077659c640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0758067660 0x7f0758069b20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f07781448a0 tx=0x7f076c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.330+0000 7f0766ffd640 1 -- 172.21.15.7:0/2846171250 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0768055a20 con 0x7f07780b9ed0 2024-03-19T01:24:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.490+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0778144c50 con 0x7f07780b9ed0 2024-03-19T01:24:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f0766ffd640 1 -- 172.21.15.7:0/2846171250 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+12982 (secure 0 0 0) 0x7f0768055320 con 0x7f07780b9ed0 2024-03-19T01:24:32.496 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:32.496 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":33,"fsid":"dafc8aaa-17d4-48ac-88de-2f029fdf817b","created":"2024-03-19T01:23:01.808901+0000","modified":"2024-03-19T01:24:28.997277+0000","last_up_change":"2024-03-19T01:23:13.395711+0000","last_in_change":"2024-03-19T01:23:04.752249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:23:14.969421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48ed6785-8fb3-4d80-9062-9fc142de0020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6801","nonce":2974347525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6803","nonce":2974347525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6807","nonce":2974347525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2974347525},{"type":"v1","addr":"172.21.15.7:6805","nonce":2974347525}]},"public_addr":"172.21.15.7:6801/2974347525","cluster_addr":"172.21.15.7:6803/2974347525","heartbeat_back_addr":"172.21.15.7:6807/2974347525","heartbeat_front_addr":"172.21.15.7:6805/2974347525","state":["exists","up"]},{"osd":1,"uuid":"e4afb6c3-9411-49f9-9eb5-2d3468ef10b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6809","nonce":1455113783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6811","nonce":1455113783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6815","nonce":1455113783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1455113783},{"type":"v1","addr":"172.21.15.7:6813","nonce":1455113783}]},"public_addr":"172.21.15.7:6809/1455113783","cluster_addr":"172.21.15.7:6811/1455113783","heartbeat_back_addr":"172.21.15.7:6815/1455113783","heartbeat_front_addr":"172.21.15.7:6813/1455113783","state":["exists","up"]},{"osd":2,"uuid":"33aff43c-f5ce-41ab-89b0-26abd458983c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6817","nonce":3939403805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6819","nonce":3939403805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6823","nonce":3939403805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3939403805},{"type":"v1","addr":"172.21.15.7:6821","nonce":3939403805}]},"public_addr":"172.21.15.7:6817/3939403805","cluster_addr":"172.21.15.7:6819/3939403805","heartbeat_back_addr":"172.21.15.7:6823/3939403805","heartbeat_front_addr":"172.21.15.7:6821/3939403805","state":["exists","up"]},{"osd":3,"uuid":"c2000d0f-e3b5-43de-8b20-2494142dbc91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6825","nonce":1705011416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6827","nonce":1705011416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6831","nonce":1705011416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1705011416},{"type":"v1","addr":"172.21.15.7:6829","nonce":1705011416}]},"public_addr":"172.21.15.7:6825/1705011416","cluster_addr":"172.21.15.7:6827/1705011416","heartbeat_back_addr":"172.21.15.7:6831/1705011416","heartbeat_front_addr":"172.21.15.7:6829/1705011416","state":["exists","up"]},{"osd":4,"uuid":"eba08682-fcb1-48ae-bd37-03af0cac45d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6801","nonce":2892054155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6807","nonce":2892054155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6819","nonce":2892054155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2892054155},{"type":"v1","addr":"172.21.15.167:6813","nonce":2892054155}]},"public_addr":"172.21.15.167:6801/2892054155","cluster_addr":"172.21.15.167:6807/2892054155","heartbeat_back_addr":"172.21.15.167:6819/2892054155","heartbeat_front_addr":"172.21.15.167:6813/2892054155","state":["exists","up"]},{"osd":5,"uuid":"d76735f6-3a76-4f1f-8817-92c4c6532a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6825","nonce":3760007162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6827","nonce":3760007162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6831","nonce":3760007162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3760007162},{"type":"v1","addr":"172.21.15.167:6829","nonce":3760007162}]},"public_addr":"172.21.15.167:6825/3760007162","cluster_addr":"172.21.15.167:6827/3760007162","heartbeat_back_addr":"172.21.15.167:6831/3760007162","heartbeat_front_addr":"172.21.15.167:6829/3760007162","state":["exists","up"]},{"osd":6,"uuid":"c5c51ab7-217b-4ec6-9c7e-d41a2a0ee352","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6803","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6806","nonce":2582461346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6809","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6812","nonce":2582461346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6821","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6822","nonce":2582461346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6815","nonce":2582461346},{"type":"v1","addr":"172.21.15.167:6818","nonce":2582461346}]},"public_addr":"172.21.15.167:6806/2582461346","cluster_addr":"172.21.15.167:6812/2582461346","heartbeat_back_addr":"172.21.15.167:6822/2582461346","heartbeat_front_addr":"172.21.15.167:6818/2582461346","state":["exists","up"]},{"osd":7,"uuid":"1bdb920f-d25e-4092-9d3f-e2cabe5e21f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6805","nonce":2393181242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6811","nonce":2393181242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6823","nonce":2393181242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2393181242},{"type":"v1","addr":"172.21.15.167:6817","nonce":2393181242}]},"public_addr":"172.21.15.167:6805/2393181242","cluster_addr":"172.21.15.167:6811/2393181242","heartbeat_back_addr":"172.21.15.167:6823/2393181242","heartbeat_front_addr":"172.21.15.167:6817/2393181242","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.159504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.218806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.269189+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.411997+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.021852+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.265722+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:09.990911+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:23:10.167092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6835/4260183338":"2024-03-20T01:24:25.747780+0000","172.21.15.7:6834/4260183338":"2024-03-20T01:24:25.747780+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:24:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0758067660 msgr2=0x7f0758069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0758067660 0x7f0758069b20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f07781448a0 tx=0x7f076c009040 comp rx=0 tx=0).stop 2024-03-19T01:24:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07780b9ed0 msgr2=0x7f07781e3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07780b9ed0 0x7f07781e3980 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f076800c390 tx=0x7f076800c850 comp rx=0 tx=0).stop 2024-03-19T01:24:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 shutdown_connections 2024-03-19T01:24:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f0758067660 0x7f0758069b20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f077814b0c0 0x7f0778138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07780b9ed0 0x7f07781e3980 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 --2- 172.21.15.7:0/2846171250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07780b4790 0x7f07781e3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 >> 172.21.15.7:0/2846171250 conn(0x7f0778072df0 msgr2=0x7f07780bae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 shutdown_connections 2024-03-19T01:24:32.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:32.494+0000 7f077cdb1640 1 -- 172.21.15.7:0/2846171250 wait complete. 2024-03-19T01:24:33.516 INFO:tasks.ceph:Scrubbing osd.0 2024-03-19T01:24:33.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-19T01:24:33.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8e3b0640 1 Processor -- start 2024-03-19T01:24:33.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8e3b0640 1 -- start start 2024-03-19T01:24:33.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8e3b0640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fcc88148390 0x7fcc880b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8e3b0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc8806c6d0 con 0x7fcc880b0430 2024-03-19T01:24:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8e3b0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc880b1580 con 0x7fcc8806c2f0 2024-03-19T01:24:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8e3b0640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fcc8806d4e0 con 0x7fcc88148390 2024-03-19T01:24:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc877fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcc880b0430 0x7fcc8814dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39572/0 (socket says 172.21.15.7:39572) 2024-03-19T01:24:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc877fe640 1 -- 172.21.15.7:0/1715556867 learned_addr learned my addr 172.21.15.7:0/1715556867 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8c94d640 1 --2- 172.21.15.7:0/1715556867 >> v2:172.21.15.167:3300/0 conn(0x7fcc88148390 0x7fcc880b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1174580980 0 0) 0x7fcc8806c6d0 con 0x7fcc880b0430 2024-03-19T01:24:33.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc74003530 con 0x7fcc880b0430 2024-03-19T01:24:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1227424634 0 0) 0x7fcc74003530 con 0x7fcc880b0430 2024-03-19T01:24:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 >> v2:172.21.15.167:3300/0 conn(0x7fcc88148390 msgr2=0x7fcc880b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 --2- 172.21.15.7:0/1715556867 >> v2:172.21.15.167:3300/0 conn(0x7fcc88148390 0x7fcc880b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 >> v1:172.21.15.167:6789/0 conn(0x7fcc8806c2f0 legacy=0x7fcc88147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc880b1cf0 con 0x7fcc880b0430 2024-03-19T01:24:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fcc70001680 con 0x7fcc880b0430 2024-03-19T01:24:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 >> v1:172.21.15.7:6789/0 conn(0x7fcc880b0430 legacy=0x7fcc8814dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc740039c0 0x7fcc74003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc74004810 0x7fcc74004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc740051d0 0x7fcc7400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8806d4e0 con 0x7fcc74004810 2024-03-19T01:24:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7400c170 con 0x7fcc740051d0 2024-03-19T01:24:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7400c370 con 0x7fcc740039c0 2024-03-19T01:24:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc877fe640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc74004810 0x7fcc74004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc87fff640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc740039c0 0x7fcc74003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8c94d640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc740051d0 0x7fcc7400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc877fe640 1 -- 172.21.15.7:0/1715556867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc740039c0 msgr2=0x7fcc74003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc877fe640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc740039c0 0x7fcc74003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc877fe640 1 -- 172.21.15.7:0/1715556867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc740051d0 msgr2=0x7fcc7400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc877fe640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc740051d0 0x7fcc7400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc877fe640 1 -- 172.21.15.7:0/1715556867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc74003530 con 0x7fcc74004810 2024-03-19T01:24:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8c94d640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc740051d0 0x7fcc7400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:33.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.654+0000 7fcc8c94d640 1 -- 172.21.15.7:0/1715556867 reap_dead start 2024-03-19T01:24:33.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc877fe640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc74004810 0x7fcc74004c90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c1c0 tx=0x7fcc7001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7416e37c450836d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:33.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc70007d00 con 0x7fcc74004810 2024-03-19T01:24:33.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc700a9020 con 0x7fcc74004810 2024-03-19T01:24:33.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc86ffd640 1 -- 172.21.15.7:0/1715556867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc70005b00 con 0x7fcc74004810 2024-03-19T01:24:33.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/1715556867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc74004810 msgr2=0x7fcc74004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc8e3b0640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc74004810 0x7fcc74004c90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c1c0 tx=0x7fcc7001cc20 comp rx=0 tx=0).stop 2024-03-19T01:24:33.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/1715556867 shutdown_connections 2024-03-19T01:24:33.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc8e3b0640 1 --2- 172.21.15.7:0/1715556867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc74004810 0x7fcc74004c90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/1715556867 >> 172.21.15.7:0/1715556867 conn(0x7fcc880b8c40 msgr2=0x7fcc880b9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:33.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/1715556867 shutdown_connections 2024-03-19T01:24:33.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/1715556867 wait complete. 2024-03-19T01:24:33.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc8e3b0640 1 Processor -- start 2024-03-19T01:24:33.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.658+0000 7fcc8e3b0640 1 -- start start 2024-03-19T01:24:33.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8e3b0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc74004810 0x7fcc8806fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8e3b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc880719c0 0x7fcc88071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8e3b0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc881e1800 0x7fcc881e1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8e3b0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcc88147c80 con 0x7fcc880719c0 2024-03-19T01:24:33.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8e3b0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcc880b2590 con 0x7fcc74004810 2024-03-19T01:24:33.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8e3b0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcc88147e00 con 0x7fcc881e1800 2024-03-19T01:24:33.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc880719c0 0x7fcc88071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc880719c0 0x7fcc88071e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33660/0 (socket says 172.21.15.7:33660) 2024-03-19T01:24:33.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc877fe640 1 -- 172.21.15.7:0/881687165 learned_addr learned my addr 172.21.15.7:0/881687165 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:33.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8c94d640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc881e1800 0x7fcc881e1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc87fff640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc74004810 0x7fcc8806fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc877fe640 1 -- 172.21.15.7:0/881687165 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc881e1800 msgr2=0x7fcc881e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc877fe640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc881e1800 0x7fcc881e1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc877fe640 1 -- 172.21.15.7:0/881687165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc74004810 msgr2=0x7fcc8806fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc877fe640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc74004810 0x7fcc8806fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc877fe640 1 -- 172.21.15.7:0/881687165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc881d3650 con 0x7fcc880719c0 2024-03-19T01:24:33.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc87fff640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc74004810 0x7fcc8806fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:33.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc877fe640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc880719c0 0x7fcc88071e20 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c1c0 tx=0x7fcc74005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:33.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc84ff9640 1 -- 172.21.15.7:0/881687165 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc700049f0 con 0x7fcc880719c0 2024-03-19T01:24:33.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc84ff9640 1 -- 172.21.15.7:0/881687165 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc70004b90 con 0x7fcc880719c0 2024-03-19T01:24:33.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc88070fc0 con 0x7fcc880719c0 2024-03-19T01:24:33.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc84ff9640 1 -- 172.21.15.7:0/881687165 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc700070d0 con 0x7fcc880719c0 2024-03-19T01:24:33.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc881d38c0 con 0x7fcc880719c0 2024-03-19T01:24:33.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc88075570 con 0x7fcc880719c0 2024-03-19T01:24:33.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc84ff9640 1 -- 172.21.15.7:0/881687165 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcc700bd060 con 0x7fcc880719c0 2024-03-19T01:24:33.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc84ff9640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc68067660 0x7fcc68069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc84ff9640 1 -- 172.21.15.7:0/881687165 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fcc7011eb60 con 0x7fcc880719c0 2024-03-19T01:24:33.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc87fff640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc68067660 0x7fcc68069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc84ff9640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7fcc6806d520 0x7fcc6806f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc84ff9640 1 -- 172.21.15.7:0/881687165 --> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc700c1ec0 con 0x7fcc6806d520 2024-03-19T01:24:33.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc84ff9640 1 -- 172.21.15.7:0/881687165 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcc7011ef10 con 0x7fcc880719c0 2024-03-19T01:24:33.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.662+0000 7fcc8c94d640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7fcc6806d520 0x7fcc6806f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.666+0000 7fcc87fff640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc68067660 0x7fcc68069b20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcc78000f90 tx=0x7fcc78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:33.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.666+0000 7fcc8c94d640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7fcc6806d520 0x7fcc6806f940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:33.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.666+0000 7fcc84ff9640 1 -- 172.21.15.7:0/881687165 <== osd.0 v2:172.21.15.7:6800/2974347525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc700c1ec0 con 0x7fcc6806d520 2024-03-19T01:24:33.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 --> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcc8806c6d0 con 0x7fcc6806d520 2024-03-19T01:24:33.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc84ff9640 1 -- 172.21.15.7:0/881687165 <== osd.0 v2:172.21.15.7:6800/2974347525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcc8806c6d0 con 0x7fcc6806d520 2024-03-19T01:24:33.682 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:33.682 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T01:24:33.682 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:33.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7fcc6806d520 msgr2=0x7fcc6806f940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6800/2974347525,v1:172.21.15.7:6801/2974347525] conn(0x7fcc6806d520 0x7fcc6806f940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc68067660 msgr2=0x7fcc68069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fcc68067660 0x7fcc68069b20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcc78000f90 tx=0x7fcc78009040 comp rx=0 tx=0).stop 2024-03-19T01:24:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc880719c0 msgr2=0x7fcc88071e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 --2- 172.21.15.7:0/881687165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc880719c0 0x7fcc88071e20 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c1c0 tx=0x7fcc74005840 comp rx=0 tx=0).stop 2024-03-19T01:24:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8c94d640 1 -- 172.21.15.7:0/881687165 reap_dead start 2024-03-19T01:24:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 shutdown_connections 2024-03-19T01:24:33.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 >> 172.21.15.7:0/881687165 conn(0x7fcc880b8c40 msgr2=0x7fcc88067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:33.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 shutdown_connections 2024-03-19T01:24:33.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.678+0000 7fcc8e3b0640 1 -- 172.21.15.7:0/881687165 wait complete. 2024-03-19T01:24:33.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-19T01:24:33.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e605e4640 1 Processor -- start 2024-03-19T01:24:33.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e605e4640 1 -- start start 2024-03-19T01:24:33.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e605e4640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f7e580b81c0 0x7f7e581486e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e605e4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e5806f1c0 0x7f7e5813f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e605e4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e580b85a0 con 0x7f7e5806f640 2024-03-19T01:24:33.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e605e4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f7e580b2710 con 0x7f7e5806f1c0 2024-03-19T01:24:33.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e605e4640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f7e58070960 con 0x7f7e580b81c0 2024-03-19T01:24:33.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e5db7f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e5806f1c0 0x7f7e5813f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e5db7f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e5806f1c0 0x7f7e5813f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33676/0 (socket says 172.21.15.7:33676) 2024-03-19T01:24:33.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e5db7f640 1 -- 172.21.15.7:0/3386010000 learned_addr learned my addr 172.21.15.7:0/3386010000 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:33.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e5e380640 1 --2- 172.21.15.7:0/3386010000 >> v2:172.21.15.167:3300/0 conn(0x7f7e580b81c0 0x7f7e581486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e5db7f640 1 -- 172.21.15.7:0/3386010000 >> v2:172.21.15.167:3300/0 conn(0x7f7e580b81c0 msgr2=0x7f7e581486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:33.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e5db7f640 1 --2- 172.21.15.7:0/3386010000 >> v2:172.21.15.167:3300/0 conn(0x7f7e580b81c0 0x7f7e581486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e5db7f640 1 -- 172.21.15.7:0/3386010000 >> v1:172.21.15.167:6789/0 conn(0x7f7e5806f640 legacy=0x7f7e58145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.814+0000 7f7e5db7f640 1 -- 172.21.15.7:0/3386010000 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e580b20c0 con 0x7f7e5806f1c0 2024-03-19T01:24:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e5db7f640 1 --2- 172.21.15.7:0/3386010000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f1c0 0x7f7e5813f0a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7e580b9950 tx=0x7f7e4801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a037a0dec8d2e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e5d37e640 1 -- 172.21.15.7:0/3386010000 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480ab680 con 0x7f7e5806f1c0 2024-03-19T01:24:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e5d37e640 1 -- 172.21.15.7:0/3386010000 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e480b0070 con 0x7f7e5806f1c0 2024-03-19T01:24:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e5d37e640 1 -- 172.21.15.7:0/3386010000 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480aba40 con 0x7f7e5806f1c0 2024-03-19T01:24:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 -- 172.21.15.7:0/3386010000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f1c0 msgr2=0x7f7e5813f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 --2- 172.21.15.7:0/3386010000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f1c0 0x7f7e5813f0a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7e580b9950 tx=0x7f7e4801c460 comp rx=0 tx=0).stop 2024-03-19T01:24:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 -- 172.21.15.7:0/3386010000 shutdown_connections 2024-03-19T01:24:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 --2- 172.21.15.7:0/3386010000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f1c0 0x7f7e5813f0a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 --2- 172.21.15.7:0/3386010000 >> v2:172.21.15.167:3300/0 conn(0x7f7e580b81c0 0x7f7e581486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 -- 172.21.15.7:0/3386010000 >> 172.21.15.7:0/3386010000 conn(0x7f7e580b5330 msgr2=0x7f7e580b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:33.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 -- 172.21.15.7:0/3386010000 shutdown_connections 2024-03-19T01:24:33.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 -- 172.21.15.7:0/3386010000 wait complete. 2024-03-19T01:24:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 Processor -- start 2024-03-19T01:24:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 -- start start 2024-03-19T01:24:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e5806f1c0 0x7f7e58144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f640 0x7f7e580b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e580b81c0 0x7f7e580b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7e58152a90 con 0x7f7e5806f640 2024-03-19T01:24:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7e58152c10 con 0x7f7e5806f1c0 2024-03-19T01:24:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e605e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7e58152d90 con 0x7f7e580b81c0 2024-03-19T01:24:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e5db7f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f640 0x7f7e580b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e5db7f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f640 0x7f7e580b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33682/0 (socket says 172.21.15.7:33682) 2024-03-19T01:24:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e5db7f640 1 -- 172.21.15.7:0/1538386854 learned_addr learned my addr 172.21.15.7:0/1538386854 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e5e380640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e5806f1c0 0x7f7e58144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.818+0000 7f7e5eb81640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e580b81c0 0x7f7e580b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e5db7f640 1 -- 172.21.15.7:0/1538386854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e580b81c0 msgr2=0x7f7e580b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e5db7f640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e580b81c0 0x7f7e580b1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e5db7f640 1 -- 172.21.15.7:0/1538386854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e5806f1c0 msgr2=0x7f7e58144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e5db7f640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e5806f1c0 0x7f7e58144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e5db7f640 1 -- 172.21.15.7:0/1538386854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e580b0890 con 0x7f7e5806f640 2024-03-19T01:24:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e5db7f640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f640 0x7f7e580b12e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7e480ad040 tx=0x7f7e4801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e4effd640 1 -- 172.21.15.7:0/1538386854 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480be5f0 con 0x7f7e5806f640 2024-03-19T01:24:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e4effd640 1 -- 172.21.15.7:0/1538386854 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e480a7070 con 0x7f7e5806f640 2024-03-19T01:24:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e4effd640 1 -- 172.21.15.7:0/1538386854 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480be9b0 con 0x7f7e5806f640 2024-03-19T01:24:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e580724a0 con 0x7f7e5806f640 2024-03-19T01:24:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5813ea90 con 0x7f7e5806f640 2024-03-19T01:24:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e58075460 con 0x7f7e5806f640 2024-03-19T01:24:33.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e4effd640 1 -- 172.21.15.7:0/1538386854 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7e480c7440 con 0x7f7e5806f640 2024-03-19T01:24:33.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e4effd640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e3c067660 0x7f7e3c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:33.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e4effd640 1 -- 172.21.15.7:0/1538386854 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f7e4811ff00 con 0x7f7e5806f640 2024-03-19T01:24:33.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e5e380640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e3c067660 0x7f7e3c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:33.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.822+0000 7f7e5e380640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e3c067660 0x7f7e3c069b20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e540079c0 tx=0x7f7e54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.826+0000 7f7e4effd640 1 -- 172.21.15.7:0/1538386854 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7e48016110 con 0x7f7e5806f640 2024-03-19T01:24:33.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.982+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f7e580b5ff0 con 0x7f7e3c067660 2024-03-19T01:24:33.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.982+0000 7f7e4effd640 1 -- 172.21.15.7:0/1538386854 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7e580b5ff0 con 0x7f7e3c067660 2024-03-19T01:24:33.986 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 0 to deep-scrub 2024-03-19T01:24:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e3c067660 msgr2=0x7f7e3c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e3c067660 0x7f7e3c069b20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e540079c0 tx=0x7f7e54008040 comp rx=0 tx=0).stop 2024-03-19T01:24:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f640 msgr2=0x7f7e580b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f640 0x7f7e580b12e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7e480ad040 tx=0x7f7e4801aca0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 shutdown_connections 2024-03-19T01:24:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f7e3c067660 0x7f7e3c069b20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e580b81c0 0x7f7e580b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e5806f640 0x7f7e580b12e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 --2- 172.21.15.7:0/1538386854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e5806f1c0 0x7f7e58144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 >> 172.21.15.7:0/1538386854 conn(0x7f7e580b5330 msgr2=0x7f7e580b6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 shutdown_connections 2024-03-19T01:24:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:33.986+0000 7f7e605e4640 1 -- 172.21.15.7:0/1538386854 wait complete. 2024-03-19T01:24:34.003 INFO:tasks.ceph:Scrubbing osd.1 2024-03-19T01:24:34.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-19T01:24:34.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f479b640 1 Processor -- start 2024-03-19T01:24:34.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f479b640 1 -- start start 2024-03-19T01:24:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f479b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f53ec06c2f0 0x7f53ec0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f479b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f53ec0b0430 0x7f53ec0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f479b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53ec06c6d0 con 0x7f53ec147c80 2024-03-19T01:24:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f479b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f53ec06d4e0 con 0x7f53ec06c2f0 2024-03-19T01:24:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f479b640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f53ec0b1580 con 0x7f53ec0b0430 2024-03-19T01:24:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f2537640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f53ec06c2f0 0x7f53ec0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f2537640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f53ec06c2f0 0x7f53ec0b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33690/0 (socket says 172.21.15.7:33690) 2024-03-19T01:24:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f2537640 1 -- 172.21.15.7:0/2756559685 learned_addr learned my addr 172.21.15.7:0/2756559685 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f1d36640 1 --2- 172.21.15.7:0/2756559685 >> v2:172.21.15.167:3300/0 conn(0x7f53ec0b0430 0x7f53ec0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f2537640 1 -- 172.21.15.7:0/2756559685 >> v2:172.21.15.167:3300/0 conn(0x7f53ec0b0430 msgr2=0x7f53ec0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f2537640 1 --2- 172.21.15.7:0/2756559685 >> v2:172.21.15.167:3300/0 conn(0x7f53ec0b0430 0x7f53ec0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f2537640 1 -- 172.21.15.7:0/2756559685 >> v1:172.21.15.167:6789/0 conn(0x7f53ec147c80 legacy=0x7f53ec1480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.118+0000 7f53f2537640 1 -- 172.21.15.7:0/2756559685 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ec0b1cf0 con 0x7f53ec06c2f0 2024-03-19T01:24:34.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f2537640 1 --2- 172.21.15.7:0/2756559685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec06c2f0 0x7f53ec0b3470 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0b6580 tx=0x7f53e801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecda3884a203e8e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f1535640 1 -- 172.21.15.7:0/2756559685 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e80a5810 con 0x7f53ec06c2f0 2024-03-19T01:24:34.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f1535640 1 -- 172.21.15.7:0/2756559685 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53e80a59b0 con 0x7f53ec06c2f0 2024-03-19T01:24:34.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f1535640 1 -- 172.21.15.7:0/2756559685 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e80a5d80 con 0x7f53ec06c2f0 2024-03-19T01:24:34.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 -- 172.21.15.7:0/2756559685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec06c2f0 msgr2=0x7f53ec0b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 --2- 172.21.15.7:0/2756559685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec06c2f0 0x7f53ec0b3470 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0b6580 tx=0x7f53e801c930 comp rx=0 tx=0).stop 2024-03-19T01:24:34.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 -- 172.21.15.7:0/2756559685 shutdown_connections 2024-03-19T01:24:34.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 --2- 172.21.15.7:0/2756559685 >> v2:172.21.15.167:3300/0 conn(0x7f53ec0b0430 0x7f53ec0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 --2- 172.21.15.7:0/2756559685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec06c2f0 0x7f53ec0b3470 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 -- 172.21.15.7:0/2756559685 >> 172.21.15.7:0/2756559685 conn(0x7f53ec0b8c40 msgr2=0x7f53ec0b9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:34.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 -- 172.21.15.7:0/2756559685 shutdown_connections 2024-03-19T01:24:34.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 -- 172.21.15.7:0/2756559685 wait complete. 2024-03-19T01:24:34.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 Processor -- start 2024-03-19T01:24:34.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 -- start start 2024-03-19T01:24:34.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f53ec06c2f0 0x7f53ec0add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec0b0430 0x7f53ec1e3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f53ec147c80 0x7f53ec1e37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec152610 con 0x7f53ec0b0430 2024-03-19T01:24:34.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec152490 con 0x7f53ec06c2f0 2024-03-19T01:24:34.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f479b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f53ec152790 con 0x7f53ec147c80 2024-03-19T01:24:34.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f1d36640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec0b0430 0x7f53ec1e3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f1d36640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec0b0430 0x7f53ec1e3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33702/0 (socket says 172.21.15.7:33702) 2024-03-19T01:24:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.122+0000 7f53f1d36640 1 -- 172.21.15.7:0/764052768 learned_addr learned my addr 172.21.15.7:0/764052768 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f2537640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f53ec06c2f0 0x7f53ec0add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f2d38640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f53ec147c80 0x7f53ec1e37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f1d36640 1 -- 172.21.15.7:0/764052768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f53ec147c80 msgr2=0x7f53ec1e37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f1d36640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f53ec147c80 0x7f53ec1e37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f1d36640 1 -- 172.21.15.7:0/764052768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f53ec06c2f0 msgr2=0x7f53ec0add10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f1d36640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f53ec06c2f0 0x7f53ec0add10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f1d36640 1 -- 172.21.15.7:0/764052768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ec0ae250 con 0x7f53ec0b0430 2024-03-19T01:24:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f2537640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f53ec06c2f0 0x7f53ec0add10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f1d36640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec0b0430 0x7f53ec1e3260 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00d800 tx=0x7f53dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 -- 172.21.15.7:0/764052768 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc00b6c0 con 0x7f53ec0b0430 2024-03-19T01:24:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 -- 172.21.15.7:0/764052768 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53dc019070 con 0x7f53ec0b0430 2024-03-19T01:24:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ec0aefb0 con 0x7f53ec0b0430 2024-03-19T01:24:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 -- 172.21.15.7:0/764052768 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc00ba60 con 0x7f53ec0b0430 2024-03-19T01:24:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ec146b70 con 0x7f53ec0b0430 2024-03-19T01:24:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f53ec075570 con 0x7f53ec0b0430 2024-03-19T01:24:34.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 -- 172.21.15.7:0/764052768 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f53dc00bc00 con 0x7f53ec0b0430 2024-03-19T01:24:34.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f53d0067660 0x7f53d0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 -- 172.21.15.7:0/764052768 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f53dc086470 con 0x7f53ec0b0430 2024-03-19T01:24:34.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f53d006d520 0x7f53d006f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f2537640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f53d0067660 0x7f53d0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 -- 172.21.15.7:0/764052768 --> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53dc001d40 con 0x7f53d006d520 2024-03-19T01:24:34.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 -- 172.21.15.7:0/764052768 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f53dc086900 con 0x7f53ec0b0430 2024-03-19T01:24:34.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f2d38640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f53d006d520 0x7f53d006f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f2537640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f53d0067660 0x7f53d0069b20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f53e80ae000 tx=0x7f53e80a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53f2d38640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f53d006d520 0x7f53d006f940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.126+0000 7f53e2ffd640 1 -- 172.21.15.7:0/764052768 <== osd.1 v2:172.21.15.7:6808/1455113783 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53dc001d40 con 0x7f53d006d520 2024-03-19T01:24:34.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 --> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f53ec09de50 con 0x7f53d006d520 2024-03-19T01:24:34.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53e2ffd640 1 -- 172.21.15.7:0/764052768 <== osd.1 v2:172.21.15.7:6808/1455113783 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f53ec09de50 con 0x7f53d006d520 2024-03-19T01:24:34.144 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:34.144 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T01:24:34.145 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:34.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f53d006d520 msgr2=0x7f53d006f940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6808/1455113783,v1:172.21.15.7:6809/1455113783] conn(0x7f53d006d520 0x7f53d006f940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f53d0067660 msgr2=0x7f53d0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f53d0067660 0x7f53d0069b20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f53e80ae000 tx=0x7f53e80a7070 comp rx=0 tx=0).stop 2024-03-19T01:24:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec0b0430 msgr2=0x7f53ec1e3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 --2- 172.21.15.7:0/764052768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ec0b0430 0x7f53ec1e3260 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00d800 tx=0x7f53dc00dcc0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f2d38640 1 -- 172.21.15.7:0/764052768 reap_dead start 2024-03-19T01:24:34.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 shutdown_connections 2024-03-19T01:24:34.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 >> 172.21.15.7:0/764052768 conn(0x7f53ec0b8c40 msgr2=0x7f53ec0b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:34.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 shutdown_connections 2024-03-19T01:24:34.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.142+0000 7f53f479b640 1 -- 172.21.15.7:0/764052768 wait complete. 2024-03-19T01:24:34.159 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-19T01:24:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4bcc5640 1 Processor -- start 2024-03-19T01:24:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4bcc5640 1 -- start start 2024-03-19T01:24:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4bcc5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fca44145d80 0x7fca44157110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4bcc5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca440b0c80 con 0x7fca44072c00 2024-03-19T01:24:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4bcc5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca440b08d0 con 0x7fca440b04f0 2024-03-19T01:24:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4bcc5640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fca440b12a0 con 0x7fca44145d80 2024-03-19T01:24:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca49260640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fca44072c00 0x7fca4414baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39586/0 (socket says 172.21.15.7:39586) 2024-03-19T01:24:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca49260640 1 -- 172.21.15.7:0/2912523184 learned_addr learned my addr 172.21.15.7:0/2912523184 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4a262640 1 --2- 172.21.15.7:0/2912523184 >> v2:172.21.15.167:3300/0 conn(0x7fca44145d80 0x7fca44157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca48a5f640 1 -- 172.21.15.7:0/2912523184 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 627979514 0 0) 0x7fca440b0c80 con 0x7fca44072c00 2024-03-19T01:24:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca48a5f640 1 -- 172.21.15.7:0/2912523184 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca30003530 con 0x7fca44072c00 2024-03-19T01:24:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca48a5f640 1 -- 172.21.15.7:0/2912523184 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 266558216 0 0) 0x7fca30003530 con 0x7fca44072c00 2024-03-19T01:24:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4a262640 1 -- 172.21.15.7:0/2912523184 >> v1:172.21.15.167:6789/0 conn(0x7fca440b04f0 legacy=0x7fca44145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4a262640 1 -- 172.21.15.7:0/2912523184 >> v1:172.21.15.7:6789/0 conn(0x7fca44072c00 legacy=0x7fca4414baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.266+0000 7fca4a262640 1 -- 172.21.15.7:0/2912523184 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca440b2130 con 0x7fca44145d80 2024-03-19T01:24:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 --2- 172.21.15.7:0/2912523184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44145d80 0x7fca44157110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009dc0 tx=0x7fca3c01c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35f55a97a64870f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca48a5f640 1 -- 172.21.15.7:0/2912523184 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c0b2070 con 0x7fca44145d80 2024-03-19T01:24:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca48a5f640 1 -- 172.21.15.7:0/2912523184 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca3c0a5550 con 0x7fca44145d80 2024-03-19T01:24:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca48a5f640 1 -- 172.21.15.7:0/2912523184 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca40001680 con 0x7fca44145d80 2024-03-19T01:24:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- 172.21.15.7:0/2912523184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44145d80 msgr2=0x7fca44157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 --2- 172.21.15.7:0/2912523184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44145d80 0x7fca44157110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009dc0 tx=0x7fca3c01c7b0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- 172.21.15.7:0/2912523184 shutdown_connections 2024-03-19T01:24:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 --2- 172.21.15.7:0/2912523184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44145d80 0x7fca44157110 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- 172.21.15.7:0/2912523184 >> 172.21.15.7:0/2912523184 conn(0x7fca440b5330 msgr2=0x7fca440b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- 172.21.15.7:0/2912523184 shutdown_connections 2024-03-19T01:24:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- 172.21.15.7:0/2912523184 wait complete. 2024-03-19T01:24:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 Processor -- start 2024-03-19T01:24:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- start start 2024-03-19T01:24:34.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44072c00 0x7fca44146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca440b04f0 0x7fca441e7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca44145d80 0x7fca441e7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fca44159e70 con 0x7fca44145d80 2024-03-19T01:24:34.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fca440aff40 con 0x7fca44072c00 2024-03-19T01:24:34.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fca44159ff0 con 0x7fca440b04f0 2024-03-19T01:24:34.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca44145d80 0x7fca441e7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca44145d80 0x7fca441e7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33714/0 (socket says 172.21.15.7:33714) 2024-03-19T01:24:34.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 -- 172.21.15.7:0/576276244 learned_addr learned my addr 172.21.15.7:0/576276244 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:34.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca49260640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca440b04f0 0x7fca441e7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca49a61640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44072c00 0x7fca44146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 -- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca440b04f0 msgr2=0x7fca441e7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca440b04f0 0x7fca441e7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 -- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44072c00 msgr2=0x7fca44146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44072c00 0x7fca44146640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 -- 172.21.15.7:0/576276244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca44144920 con 0x7fca44145d80 2024-03-19T01:24:34.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca49260640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca440b04f0 0x7fca441e7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:34.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca49a61640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44072c00 0x7fca44146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:34.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4a262640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca44145d80 0x7fca441e7fe0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fca3c01a3e0 tx=0x7fca3c019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca3a7fc640 1 -- 172.21.15.7:0/576276244 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c0b2070 con 0x7fca44145d80 2024-03-19T01:24:34.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca3a7fc640 1 -- 172.21.15.7:0/576276244 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca3c0bc070 con 0x7fca44145d80 2024-03-19T01:24:34.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca440703c0 con 0x7fca44145d80 2024-03-19T01:24:34.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca3a7fc640 1 -- 172.21.15.7:0/576276244 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c0a5d10 con 0x7fca44145d80 2024-03-19T01:24:34.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.270+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca440720f0 con 0x7fca44145d80 2024-03-19T01:24:34.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.274+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca44144c50 con 0x7fca44145d80 2024-03-19T01:24:34.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.274+0000 7fca3a7fc640 1 -- 172.21.15.7:0/576276244 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fca3c004040 con 0x7fca44145d80 2024-03-19T01:24:34.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.274+0000 7fca3a7fc640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fca28067660 0x7fca28069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.274+0000 7fca3a7fc640 1 -- 172.21.15.7:0/576276244 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fca3c1200f0 con 0x7fca44145d80 2024-03-19T01:24:34.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.274+0000 7fca49a61640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fca28067660 0x7fca28069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.274+0000 7fca49a61640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fca28067660 0x7fca28069b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fca34000f70 tx=0x7fca34009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.278+0000 7fca3a7fc640 1 -- 172.21.15.7:0/576276244 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fca3c0164a0 con 0x7fca44145d80 2024-03-19T01:24:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fca44075460 con 0x7fca28067660 2024-03-19T01:24:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca3a7fc640 1 -- 172.21.15.7:0/576276244 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fca44075460 con 0x7fca28067660 2024-03-19T01:24:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 1 to deep-scrub 2024-03-19T01:24:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fca28067660 msgr2=0x7fca28069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fca28067660 0x7fca28069b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fca34000f70 tx=0x7fca34009040 comp rx=0 tx=0).stop 2024-03-19T01:24:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca44145d80 msgr2=0x7fca441e7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca44145d80 0x7fca441e7fe0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fca3c01a3e0 tx=0x7fca3c019550 comp rx=0 tx=0).stop 2024-03-19T01:24:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 shutdown_connections 2024-03-19T01:24:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fca28067660 0x7fca28069b20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca440b04f0 0x7fca441e7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca44145d80 0x7fca441e7fe0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 --2- 172.21.15.7:0/576276244 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44072c00 0x7fca44146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 >> 172.21.15.7:0/576276244 conn(0x7fca440b5330 msgr2=0x7fca440b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 shutdown_connections 2024-03-19T01:24:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.438+0000 7fca4bcc5640 1 -- 172.21.15.7:0/576276244 wait complete. 2024-03-19T01:24:34.460 INFO:tasks.ceph:Scrubbing osd.2 2024-03-19T01:24:34.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-19T01:24:34.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9c21a4640 1 Processor -- start 2024-03-19T01:24:34.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9c21a4640 1 -- start start 2024-03-19T01:24:34.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9c21a4640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fb9bc0b2af0 0x7fb9bc147c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9c21a4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc1481c0 0x7fb9bc14eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9c21a4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9bc0b0230 con 0x7fb9bc0b47e0 2024-03-19T01:24:34.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9c21a4640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fb9bc0ad9d0 con 0x7fb9bc0b2af0 2024-03-19T01:24:34.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9c21a4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb9bc0b4bc0 con 0x7fb9bc1481c0 2024-03-19T01:24:34.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9bb7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb9bc0b47e0 0x7fb9bc0aed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39600/0 (socket says 172.21.15.7:39600) 2024-03-19T01:24:34.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9bb7fe640 1 -- 172.21.15.7:0/1209786905 learned_addr learned my addr 172.21.15.7:0/1209786905 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:34.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9bbfff640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc1481c0 0x7fb9bc14eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9baffd640 1 --2- 172.21.15.7:0/1209786905 >> v2:172.21.15.167:3300/0 conn(0x7fb9bc0b2af0 0x7fb9bc147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1851793493 0 0) 0x7fb9bc0b0230 con 0x7fb9bc0b47e0 2024-03-19T01:24:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9a4003530 con 0x7fb9bc0b47e0 2024-03-19T01:24:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1102090620 0 0) 0x7fb9a4003530 con 0x7fb9bc0b47e0 2024-03-19T01:24:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc1481c0 msgr2=0x7fb9bc14eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9ba7fc640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc1481c0 0x7fb9bc14eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 >> v2:172.21.15.167:3300/0 conn(0x7fb9bc0b2af0 msgr2=0x7fb9bc147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9ba7fc640 1 --2- 172.21.15.7:0/1209786905 >> v2:172.21.15.167:3300/0 conn(0x7fb9bc0b2af0 0x7fb9bc147c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc0af4a0 con 0x7fb9bc0b47e0 2024-03-19T01:24:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.590+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fb9ac001a00 con 0x7fb9bc0b47e0 2024-03-19T01:24:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 >> v1:172.21.15.7:6789/0 conn(0x7fb9bc0b47e0 legacy=0x7fb9bc0aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 0x7fb9a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc1481c0 0x7fb9a4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9a4004d50 0x7fb9a40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc0ad9d0 con 0x7fb9a40039c0 2024-03-19T01:24:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc0b4bc0 con 0x7fb9a4004d50 2024-03-19T01:24:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a400b9d0 con 0x7fb9bc1481c0 2024-03-19T01:24:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bb7fe640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 0x7fb9a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bbfff640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9a4004d50 0x7fb9a40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9baffd640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc1481c0 0x7fb9a4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bb7fe640 1 -- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc1481c0 msgr2=0x7fb9a4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bb7fe640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc1481c0 0x7fb9a4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bb7fe640 1 -- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9a4004d50 msgr2=0x7fb9a40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bb7fe640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9a4004d50 0x7fb9a40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bb7fe640 1 -- 172.21.15.7:0/1209786905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a4003530 con 0x7fb9a40039c0 2024-03-19T01:24:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bbfff640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9a4004d50 0x7fb9a40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9baffd640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc1481c0 0x7fb9a4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bb7fe640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 0x7fb9a4003de0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00a7d0 tx=0x7fb9ac01d2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453f052ac3dd8b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac00a3f0 con 0x7fb9a40039c0 2024-03-19T01:24:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac0ad650 con 0x7fb9a40039c0 2024-03-19T01:24:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9ba7fc640 1 -- 172.21.15.7:0/1209786905 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0ada40 con 0x7fb9a40039c0 2024-03-19T01:24:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1209786905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 msgr2=0x7fb9a4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9c21a4640 1 --2- 172.21.15.7:0/1209786905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 0x7fb9a4003de0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00a7d0 tx=0x7fb9ac01d2c0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9bbfff640 1 -- 172.21.15.7:0/1209786905 reap_dead start 2024-03-19T01:24:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1209786905 shutdown_connections 2024-03-19T01:24:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1209786905 >> 172.21.15.7:0/1209786905 conn(0x7fb9bc0b8c40 msgr2=0x7fb9bc0b9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1209786905 shutdown_connections 2024-03-19T01:24:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.594+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1209786905 wait complete. 2024-03-19T01:24:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 Processor -- start 2024-03-19T01:24:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 -- start start 2024-03-19T01:24:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 0x7fb9bc09adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc09bb20 0x7fb9bc144a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9bc1e0c00 0x7fb9bc0b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc1485d0 con 0x7fb9a40039c0 2024-03-19T01:24:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc0b0c10 con 0x7fb9bc1e0c00 2024-03-19T01:24:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb9bc148750 con 0x7fb9bc09bb20 2024-03-19T01:24:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 0x7fb9bc09adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 0x7fb9bc09adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33722/0 (socket says 172.21.15.7:33722) 2024-03-19T01:24:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bb7fe640 1 -- 172.21.15.7:0/1301858547 learned_addr learned my addr 172.21.15.7:0/1301858547 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9baffd640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc09bb20 0x7fb9bc144a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bbfff640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9bc1e0c00 0x7fb9bc0b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bb7fe640 1 -- 172.21.15.7:0/1301858547 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc09bb20 msgr2=0x7fb9bc144a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bb7fe640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc09bb20 0x7fb9bc144a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bb7fe640 1 -- 172.21.15.7:0/1301858547 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9bc1e0c00 msgr2=0x7fb9bc0b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bb7fe640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9bc1e0c00 0x7fb9bc0b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bb7fe640 1 -- 172.21.15.7:0/1301858547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc141560 con 0x7fb9a40039c0 2024-03-19T01:24:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9baffd640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9bc09bb20 0x7fb9bc144a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9bb7fe640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 0x7fb9bc09adc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac000f90 tx=0x7fb9a4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9a3fff640 1 -- 172.21.15.7:0/1301858547 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0bc070 con 0x7fb9a40039c0 2024-03-19T01:24:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9a3fff640 1 -- 172.21.15.7:0/1301858547 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac004d70 con 0x7fb9a40039c0 2024-03-19T01:24:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9bc09c070 con 0x7fb9a40039c0 2024-03-19T01:24:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9a3fff640 1 -- 172.21.15.7:0/1301858547 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0b2040 con 0x7fb9a40039c0 2024-03-19T01:24:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9bc13e9c0 con 0x7fb9a40039c0 2024-03-19T01:24:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9bc075570 con 0x7fb9a40039c0 2024-03-19T01:24:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.598+0000 7fb9a3fff640 1 -- 172.21.15.7:0/1301858547 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb9ac0bf070 con 0x7fb9a40039c0 2024-03-19T01:24:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9a3fff640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb9a8067690 0x7fb9a8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9a3fff640 1 -- 172.21.15.7:0/1301858547 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fb9ac120050 con 0x7fb9a40039c0 2024-03-19T01:24:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9baffd640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb9a8067690 0x7fb9a8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9a3fff640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7fb9a806d550 0x7fb9a806f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9a3fff640 1 -- 172.21.15.7:0/1301858547 --> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9ac007ad0 con 0x7fb9a806d550 2024-03-19T01:24:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9a3fff640 1 -- 172.21.15.7:0/1301858547 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb9ac1204e0 con 0x7fb9a40039c0 2024-03-19T01:24:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9bbfff640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7fb9a806d550 0x7fb9a806f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9baffd640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb9a8067690 0x7fb9a8069b50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0b22e0 tx=0x7fb9b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9bbfff640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7fb9a806d550 0x7fb9a806f970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.602+0000 7fb9a3fff640 1 -- 172.21.15.7:0/1301858547 <== osd.2 v2:172.21.15.7:6816/3939403805 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9ac007ad0 con 0x7fb9a806d550 2024-03-19T01:24:34.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.614+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 --> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb9bc09e180 con 0x7fb9a806d550 2024-03-19T01:24:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.614+0000 7fb9a3fff640 1 -- 172.21.15.7:0/1301858547 <== osd.2 v2:172.21.15.7:6816/3939403805 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb9bc09e180 con 0x7fb9a806d550 2024-03-19T01:24:34.619 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:34.619 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T01:24:34.619 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:34.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7fb9a806d550 msgr2=0x7fb9a806f970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6816/3939403805,v1:172.21.15.7:6817/3939403805] conn(0x7fb9a806d550 0x7fb9a806f970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb9a8067690 msgr2=0x7fb9a8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb9a8067690 0x7fb9a8069b50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0b22e0 tx=0x7fb9b0008040 comp rx=0 tx=0).stop 2024-03-19T01:24:34.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 msgr2=0x7fb9bc09adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 --2- 172.21.15.7:0/1301858547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9a40039c0 0x7fb9bc09adc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac000f90 tx=0x7fb9a4005840 comp rx=0 tx=0).stop 2024-03-19T01:24:34.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9bbfff640 1 -- 172.21.15.7:0/1301858547 reap_dead start 2024-03-19T01:24:34.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 shutdown_connections 2024-03-19T01:24:34.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 >> 172.21.15.7:0/1301858547 conn(0x7fb9bc0b8c40 msgr2=0x7fb9bc071570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:34.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 shutdown_connections 2024-03-19T01:24:34.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.618+0000 7fb9c21a4640 1 -- 172.21.15.7:0/1301858547 wait complete. 2024-03-19T01:24:34.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-19T01:24:34.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f39251f3640 1 Processor -- start 2024-03-19T01:24:34.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f39251f3640 1 -- start start 2024-03-19T01:24:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f39251f3640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f39200bd660 0x7f392014a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f39251f3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f392014ae60 0x7f3920151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f39251f3640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39200b6ce0 con 0x7f39200b4570 2024-03-19T01:24:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f39251f3640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f39200b4950 con 0x7f392014ae60 2024-03-19T01:24:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f39251f3640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f39200b51c0 con 0x7f39200bd660 2024-03-19T01:24:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391ed9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f39200b4570 0x7f3920140fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39602/0 (socket says 172.21.15.7:39602) 2024-03-19T01:24:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391ed9d640 1 -- 172.21.15.7:0/134791314 learned_addr learned my addr 172.21.15.7:0/134791314 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391f59e640 1 --2- 172.21.15.7:0/134791314 >> v2:172.21.15.7:3300/0 conn(0x7f392014ae60 0x7f3920151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391dd9b640 1 -- 172.21.15.7:0/134791314 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1897112964 0 0) 0x7f39200b6ce0 con 0x7f39200b4570 2024-03-19T01:24:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391e59c640 1 --2- 172.21.15.7:0/134791314 >> v2:172.21.15.167:3300/0 conn(0x7f39200bd660 0x7f392014a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391dd9b640 1 -- 172.21.15.7:0/134791314 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3908003530 con 0x7f39200b4570 2024-03-19T01:24:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391f59e640 1 -- 172.21.15.7:0/134791314 >> v2:172.21.15.167:3300/0 conn(0x7f39200bd660 msgr2=0x7f392014a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391f59e640 1 --2- 172.21.15.7:0/134791314 >> v2:172.21.15.167:3300/0 conn(0x7f39200bd660 0x7f392014a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391f59e640 1 -- 172.21.15.7:0/134791314 >> v1:172.21.15.7:6789/0 conn(0x7f39200b4570 legacy=0x7f3920140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391f59e640 1 -- 172.21.15.7:0/134791314 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39200b5480 con 0x7f392014ae60 2024-03-19T01:24:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391e59c640 1 --2- 172.21.15.7:0/134791314 >> v2:172.21.15.167:3300/0 conn(0x7f39200bd660 0x7f392014a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:34.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391f59e640 1 --2- 172.21.15.7:0/134791314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 0x7f3920151cb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f39200b3fd0 tx=0x7f391801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e46b40e0e94add server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391dd9b640 1 -- 172.21.15.7:0/134791314 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39180a66e0 con 0x7f392014ae60 2024-03-19T01:24:34.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391dd9b640 1 -- 172.21.15.7:0/134791314 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39180a6880 con 0x7f392014ae60 2024-03-19T01:24:34.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.746+0000 7f391dd9b640 1 -- 172.21.15.7:0/134791314 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39180a6c30 con 0x7f392014ae60 2024-03-19T01:24:34.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- 172.21.15.7:0/134791314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 msgr2=0x7f3920151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 --2- 172.21.15.7:0/134791314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 0x7f3920151cb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f39200b3fd0 tx=0x7f391801c810 comp rx=0 tx=0).stop 2024-03-19T01:24:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- 172.21.15.7:0/134791314 shutdown_connections 2024-03-19T01:24:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 --2- 172.21.15.7:0/134791314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 0x7f3920151cb0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 --2- 172.21.15.7:0/134791314 >> v2:172.21.15.167:3300/0 conn(0x7f39200bd660 0x7f392014a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- 172.21.15.7:0/134791314 >> 172.21.15.7:0/134791314 conn(0x7f3920072df0 msgr2=0x7f3920073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- 172.21.15.7:0/134791314 shutdown_connections 2024-03-19T01:24:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- 172.21.15.7:0/134791314 wait complete. 2024-03-19T01:24:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 Processor -- start 2024-03-19T01:24:34.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- start start 2024-03-19T01:24:34.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39200b4570 0x7f39200bc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 0x7f3920142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3920139a30 0x7f3920137d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f39200b93b0 con 0x7f392014ae60 2024-03-19T01:24:34.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f39200b9230 con 0x7f39200b4570 2024-03-19T01:24:34.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3920154a10 con 0x7f3920139a30 2024-03-19T01:24:34.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f391e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 0x7f3920142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f391e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 0x7f3920142430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33740/0 (socket says 172.21.15.7:33740) 2024-03-19T01:24:34.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f391e59c640 1 -- 172.21.15.7:0/3576821682 learned_addr learned my addr 172.21.15.7:0/3576821682 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:34.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f391e59c640 1 -- 172.21.15.7:0/3576821682 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3920139a30 msgr2=0x7f3920137d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f391e59c640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3920139a30 0x7f3920137d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f391e59c640 1 -- 172.21.15.7:0/3576821682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39200b4570 msgr2=0x7f39200bc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f391e59c640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39200b4570 0x7f39200bc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f391e59c640 1 -- 172.21.15.7:0/3576821682 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3920144480 con 0x7f392014ae60 2024-03-19T01:24:34.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f391e59c640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 0x7f3920142430 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f391400c7f0 tx=0x7f391400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39077fe640 1 -- 172.21.15.7:0/3576821682 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914015070 con 0x7f392014ae60 2024-03-19T01:24:34.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39077fe640 1 -- 172.21.15.7:0/3576821682 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3914007a30 con 0x7f392014ae60 2024-03-19T01:24:34.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39201437c0 con 0x7f392014ae60 2024-03-19T01:24:34.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39077fe640 1 -- 172.21.15.7:0/3576821682 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914007e20 con 0x7f392014ae60 2024-03-19T01:24:34.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3920144720 con 0x7f392014ae60 2024-03-19T01:24:34.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.750+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3920075460 con 0x7f392014ae60 2024-03-19T01:24:34.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.754+0000 7f39077fe640 1 -- 172.21.15.7:0/3576821682 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3914018070 con 0x7f392014ae60 2024-03-19T01:24:34.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.754+0000 7f39077fe640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f390c067660 0x7f390c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:34.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.754+0000 7f39077fe640 1 -- 172.21.15.7:0/3576821682 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f39140854a0 con 0x7f392014ae60 2024-03-19T01:24:34.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.754+0000 7f391ed9d640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f390c067660 0x7f390c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:34.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.754+0000 7f391ed9d640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f390c067660 0x7f390c069b20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3910007fd0 tx=0x7f3910007900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:34.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.758+0000 7f39077fe640 1 -- 172.21.15.7:0/3576821682 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f39140564b0 con 0x7f392014ae60 2024-03-19T01:24:34.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f3920137890 con 0x7f390c067660 2024-03-19T01:24:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39077fe640 1 -- 172.21.15.7:0/3576821682 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3920137890 con 0x7f390c067660 2024-03-19T01:24:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 2 to deep-scrub 2024-03-19T01:24:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f390c067660 msgr2=0x7f390c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f390c067660 0x7f390c069b20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3910007fd0 tx=0x7f3910007900 comp rx=0 tx=0).stop 2024-03-19T01:24:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 msgr2=0x7f3920142430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 0x7f3920142430 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f391400c7f0 tx=0x7f391400ccb0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 shutdown_connections 2024-03-19T01:24:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f390c067660 0x7f390c069b20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3920139a30 0x7f3920137d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f392014ae60 0x7f3920142430 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 --2- 172.21.15.7:0/3576821682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39200b4570 0x7f39200bc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 >> 172.21.15.7:0/3576821682 conn(0x7f3920072df0 msgr2=0x7f3920145620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:34.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 shutdown_connections 2024-03-19T01:24:34.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:34.914+0000 7f39251f3640 1 -- 172.21.15.7:0/3576821682 wait complete. 2024-03-19T01:24:34.935 INFO:tasks.ceph:Scrubbing osd.3 2024-03-19T01:24:34.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-19T01:24:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.046+0000 7f23dc8cb640 1 Processor -- start 2024-03-19T01:24:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.046+0000 7f23dc8cb640 1 -- start start 2024-03-19T01:24:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23dc8cb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f23d806c2f0 0x7f23d80b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23dc8cb640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f23d80b0430 0x7f23d80b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23dc8cb640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d806c6d0 con 0x7f23d8147c80 2024-03-19T01:24:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23dc8cb640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f23d806d4e0 con 0x7f23d80b0430 2024-03-19T01:24:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23dc8cb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f23d80b1580 con 0x7f23d806c2f0 2024-03-19T01:24:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d5d9b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f23d80b0430 0x7f23d80b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d6d9d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f23d8147c80 0x7f23d8148110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:59300/0 (socket says 172.21.15.7:59300) 2024-03-19T01:24:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d6d9d640 1 -- 172.21.15.7:0/3250477196 learned_addr learned my addr 172.21.15.7:0/3250477196 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d659c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f23d806c2f0 0x7f23d80b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d5d9b640 1 -- 172.21.15.7:0/3250477196 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f23d806c2f0 msgr2=0x7f23d80b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d5d9b640 1 --2- 172.21.15.7:0/3250477196 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f23d806c2f0 0x7f23d80b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d5d9b640 1 -- 172.21.15.7:0/3250477196 >> v1:172.21.15.167:6789/0 conn(0x7f23d8147c80 legacy=0x7f23d8148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d5d9b640 1 -- 172.21.15.7:0/3250477196 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d80b1cf0 con 0x7f23d80b0430 2024-03-19T01:24:35.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d5d9b640 1 --2- 172.21.15.7:0/3250477196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f23d80b0430 0x7f23d80b2ab0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0097c0 tx=0x7f23cc01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7eda9ad6886e9184 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d559a640 1 -- 172.21.15.7:0/3250477196 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0b1070 con 0x7f23d80b0430 2024-03-19T01:24:35.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d559a640 1 -- 172.21.15.7:0/3250477196 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23cc0a8050 con 0x7f23d80b0430 2024-03-19T01:24:35.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.050+0000 7f23d559a640 1 -- 172.21.15.7:0/3250477196 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0ac860 con 0x7f23d80b0430 2024-03-19T01:24:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- 172.21.15.7:0/3250477196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f23d80b0430 msgr2=0x7f23d80b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 --2- 172.21.15.7:0/3250477196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f23d80b0430 0x7f23d80b2ab0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0097c0 tx=0x7f23cc01c3e0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- 172.21.15.7:0/3250477196 shutdown_connections 2024-03-19T01:24:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 --2- 172.21.15.7:0/3250477196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f23d80b0430 0x7f23d80b2ab0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 --2- 172.21.15.7:0/3250477196 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f23d806c2f0 0x7f23d80b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- 172.21.15.7:0/3250477196 >> 172.21.15.7:0/3250477196 conn(0x7f23d80b8c40 msgr2=0x7f23d80b9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- 172.21.15.7:0/3250477196 shutdown_connections 2024-03-19T01:24:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- 172.21.15.7:0/3250477196 wait complete. 2024-03-19T01:24:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 Processor -- start 2024-03-19T01:24:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- start start 2024-03-19T01:24:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f23d806c2f0 0x7f23d809b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f23d80b0430 0x7f23d81dd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d8147c80 0x7f23d809ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f23d8152610 con 0x7f23d8147c80 2024-03-19T01:24:35.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f23d8152490 con 0x7f23d80b0430 2024-03-19T01:24:35.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f23d8152790 con 0x7f23d806c2f0 2024-03-19T01:24:35.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d8147c80 0x7f23d809ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d8147c80 0x7f23d809ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33752/0 (socket says 172.21.15.7:33752) 2024-03-19T01:24:35.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d6d9d640 1 -- 172.21.15.7:0/1368179744 learned_addr learned my addr 172.21.15.7:0/1368179744 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:35.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d5d9b640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f23d80b0430 0x7f23d81dd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d6d9d640 1 -- 172.21.15.7:0/1368179744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f23d806c2f0 msgr2=0x7f23d809b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d659c640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f23d806c2f0 0x7f23d809b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d6d9d640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f23d806c2f0 0x7f23d809b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d6d9d640 1 -- 172.21.15.7:0/1368179744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f23d80b0430 msgr2=0x7f23d81dd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d6d9d640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f23d80b0430 0x7f23d81dd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d6d9d640 1 -- 172.21.15.7:0/1368179744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d8146bd0 con 0x7f23d8147c80 2024-03-19T01:24:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d5d9b640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f23d80b0430 0x7f23d81dd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23d6d9d640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d8147c80 0x7f23d809ced0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f23c800c3f0 tx=0x7f23c800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23c6ffd640 1 -- 172.21.15.7:0/1368179744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c8017070 con 0x7f23d8147c80 2024-03-19T01:24:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23c6ffd640 1 -- 172.21.15.7:0/1368179744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23c8005cf0 con 0x7f23d8147c80 2024-03-19T01:24:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23d809d970 con 0x7f23d8147c80 2024-03-19T01:24:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23c6ffd640 1 -- 172.21.15.7:0/1368179744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c80072f0 con 0x7f23d8147c80 2024-03-19T01:24:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23d809bd60 con 0x7f23d8147c80 2024-03-19T01:24:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.054+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f23d8075570 con 0x7f23d8147c80 2024-03-19T01:24:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23c6ffd640 1 -- 172.21.15.7:0/1368179744 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f23c801a070 con 0x7f23d8147c80 2024-03-19T01:24:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23c6ffd640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f23b8067660 0x7f23b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23c6ffd640 1 -- 172.21.15.7:0/1368179744 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f23c8084fa0 con 0x7f23d8147c80 2024-03-19T01:24:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23d659c640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f23b8067660 0x7f23b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23c6ffd640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f23b806d520 0x7f23b806f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23c6ffd640 1 -- 172.21.15.7:0/1368179744 --> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23b806fff0 con 0x7f23b806d520 2024-03-19T01:24:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23c6ffd640 1 -- 172.21.15.7:0/1368179744 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f23c8085430 con 0x7f23d8147c80 2024-03-19T01:24:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23d5d9b640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f23b806d520 0x7f23b806f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23d659c640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f23b8067660 0x7f23b8069b20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009fd0 tx=0x7f23d000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23d5d9b640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f23b806d520 0x7f23b806f940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.058+0000 7f23c6ffd640 1 -- 172.21.15.7:0/1368179744 <== osd.3 v2:172.21.15.7:6824/1705011416 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23b806fff0 con 0x7f23b806d520 2024-03-19T01:24:35.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 --> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f23d81dde10 con 0x7f23b806d520 2024-03-19T01:24:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23c6ffd640 1 -- 172.21.15.7:0/1368179744 <== osd.3 v2:172.21.15.7:6824/1705011416 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f23d81dde10 con 0x7f23b806d520 2024-03-19T01:24:35.074 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:35.074 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T01:24:35.074 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f23b806d520 msgr2=0x7f23b806f940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6824/1705011416,v1:172.21.15.7:6825/1705011416] conn(0x7f23b806d520 0x7f23b806f940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f23b8067660 msgr2=0x7f23b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f23b8067660 0x7f23b8069b20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009fd0 tx=0x7f23d000b040 comp rx=0 tx=0).stop 2024-03-19T01:24:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d8147c80 msgr2=0x7f23d809ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 --2- 172.21.15.7:0/1368179744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d8147c80 0x7f23d809ced0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f23c800c3f0 tx=0x7f23c800c8b0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23d6d9d640 1 -- 172.21.15.7:0/1368179744 reap_dead start 2024-03-19T01:24:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 shutdown_connections 2024-03-19T01:24:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 >> 172.21.15.7:0/1368179744 conn(0x7f23d80b8c40 msgr2=0x7f23d80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 shutdown_connections 2024-03-19T01:24:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.070+0000 7f23dc8cb640 1 -- 172.21.15.7:0/1368179744 wait complete. 2024-03-19T01:24:35.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-19T01:24:35.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93e500b640 1 Processor -- start 2024-03-19T01:24:35.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93e500b640 1 -- start start 2024-03-19T01:24:35.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93e500b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f93e00b05f0 0x7f93e01480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93e500b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f93e013dfe0 0x7f93e013e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93e500b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e0074770 con 0x7f93e0074390 2024-03-19T01:24:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93e500b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f93e0070160 con 0x7f93e00b05f0 2024-03-19T01:24:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93e500b640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f93e006fe80 con 0x7f93e013dfe0 2024-03-19T01:24:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93de59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f93e00b05f0 0x7f93e01480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93de59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f93e00b05f0 0x7f93e01480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33768/0 (socket says 172.21.15.7:33768) 2024-03-19T01:24:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.194+0000 7f93de59c640 1 -- 172.21.15.7:0/1033242446 learned_addr learned my addr 172.21.15.7:0/1033242446 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93df59e640 1 --2- 172.21.15.7:0/1033242446 >> v2:172.21.15.167:3300/0 conn(0x7f93e013dfe0 0x7f93e013e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93de59c640 1 -- 172.21.15.7:0/1033242446 >> v2:172.21.15.167:3300/0 conn(0x7f93e013dfe0 msgr2=0x7f93e013e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93de59c640 1 --2- 172.21.15.7:0/1033242446 >> v2:172.21.15.167:3300/0 conn(0x7f93e013dfe0 0x7f93e013e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93de59c640 1 -- 172.21.15.7:0/1033242446 >> v1:172.21.15.167:6789/0 conn(0x7f93e0074390 legacy=0x7f93e00b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93de59c640 1 -- 172.21.15.7:0/1033242446 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e01475e0 con 0x7f93e00b05f0 2024-03-19T01:24:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93de59c640 1 --2- 172.21.15.7:0/1033242446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 0x7f93e01480c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f93e0074f20 tx=0x7f93d401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ab991f66f7249a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93ddd9b640 1 -- 172.21.15.7:0/1033242446 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40b2070 con 0x7f93e00b05f0 2024-03-19T01:24:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93ddd9b640 1 -- 172.21.15.7:0/1033242446 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d40ad680 con 0x7f93e00b05f0 2024-03-19T01:24:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93ddd9b640 1 -- 172.21.15.7:0/1033242446 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40ada50 con 0x7f93e00b05f0 2024-03-19T01:24:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93e500b640 1 -- 172.21.15.7:0/1033242446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 msgr2=0x7f93e01480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93e500b640 1 --2- 172.21.15.7:0/1033242446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 0x7f93e01480c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f93e0074f20 tx=0x7f93d401c490 comp rx=0 tx=0).stop 2024-03-19T01:24:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93e500b640 1 -- 172.21.15.7:0/1033242446 shutdown_connections 2024-03-19T01:24:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93e500b640 1 --2- 172.21.15.7:0/1033242446 >> v2:172.21.15.167:3300/0 conn(0x7f93e013dfe0 0x7f93e013e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93e500b640 1 --2- 172.21.15.7:0/1033242446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 0x7f93e01480c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93e500b640 1 -- 172.21.15.7:0/1033242446 >> 172.21.15.7:0/1033242446 conn(0x7f93e00b5bf0 msgr2=0x7f93e00b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93e500b640 1 -- 172.21.15.7:0/1033242446 shutdown_connections 2024-03-19T01:24:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.198+0000 7f93e500b640 1 -- 172.21.15.7:0/1033242446 wait complete. 2024-03-19T01:24:35.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 Processor -- start 2024-03-19T01:24:35.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 -- start start 2024-03-19T01:24:35.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93e0074390 0x7f93e0144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 0x7f93e013ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93e013dfe0 0x7f93e0134310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f93e0151520 con 0x7f93e00b05f0 2024-03-19T01:24:35.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01513a0 con 0x7f93e0074390 2024-03-19T01:24:35.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f93e01516a0 con 0x7f93e013dfe0 2024-03-19T01:24:35.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93de59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 0x7f93e013ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93de59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 0x7f93e013ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33778/0 (socket says 172.21.15.7:33778) 2024-03-19T01:24:35.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93de59c640 1 -- 172.21.15.7:0/1582565978 learned_addr learned my addr 172.21.15.7:0/1582565978 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:35.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93df59e640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93e013dfe0 0x7f93e0134310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93ded9d640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93e0074390 0x7f93e0144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93de59c640 1 -- 172.21.15.7:0/1582565978 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93e013dfe0 msgr2=0x7f93e0134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93de59c640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93e013dfe0 0x7f93e0134310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93de59c640 1 -- 172.21.15.7:0/1582565978 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93e0074390 msgr2=0x7f93e0144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93de59c640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93e0074390 0x7f93e0144620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93de59c640 1 -- 172.21.15.7:0/1582565978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e013a310 con 0x7f93e00b05f0 2024-03-19T01:24:35.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93ded9d640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93e0074390 0x7f93e0144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:35.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93de59c640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 0x7f93e013ec80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f93d401a0c0 tx=0x7f93d401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93c77fe640 1 -- 172.21.15.7:0/1582565978 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40b2040 con 0x7f93e00b05f0 2024-03-19T01:24:35.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93c77fe640 1 -- 172.21.15.7:0/1582565978 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d40ada50 con 0x7f93e00b05f0 2024-03-19T01:24:35.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e0140060 con 0x7f93e00b05f0 2024-03-19T01:24:35.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93c77fe640 1 -- 172.21.15.7:0/1582565978 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40ade20 con 0x7f93e00b05f0 2024-03-19T01:24:35.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e01396d0 con 0x7f93e00b05f0 2024-03-19T01:24:35.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e0075460 con 0x7f93e00b05f0 2024-03-19T01:24:35.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93c77fe640 1 -- 172.21.15.7:0/1582565978 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f93d40bf030 con 0x7f93e00b05f0 2024-03-19T01:24:35.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93c77fe640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f93cc067660 0x7f93cc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93c77fe640 1 -- 172.21.15.7:0/1582565978 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f93d411ebd0 con 0x7f93e00b05f0 2024-03-19T01:24:35.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.202+0000 7f93ded9d640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f93cc067660 0x7f93cc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.206+0000 7f93ded9d640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f93cc067660 0x7f93cc069b20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f93d8007fd0 tx=0x7f93d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.210+0000 7f93c77fe640 1 -- 172.21.15.7:0/1582565978 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f93d4016110 con 0x7f93e00b05f0 2024-03-19T01:24:35.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.362+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f93e00b54c0 con 0x7f93cc067660 2024-03-19T01:24:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.362+0000 7f93c77fe640 1 -- 172.21.15.7:0/1582565978 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f93e00b54c0 con 0x7f93cc067660 2024-03-19T01:24:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 3 to deep-scrub 2024-03-19T01:24:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f93cc067660 msgr2=0x7f93cc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f93cc067660 0x7f93cc069b20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f93d8007fd0 tx=0x7f93d8009040 comp rx=0 tx=0).stop 2024-03-19T01:24:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 msgr2=0x7f93e013ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 0x7f93e013ec80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f93d401a0c0 tx=0x7f93d401acd0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 shutdown_connections 2024-03-19T01:24:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f93cc067660 0x7f93cc069b20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93e013dfe0 0x7f93e0134310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93e00b05f0 0x7f93e013ec80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 --2- 172.21.15.7:0/1582565978 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93e0074390 0x7f93e0144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 >> 172.21.15.7:0/1582565978 conn(0x7f93e00b5bf0 msgr2=0x7f93e00b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 shutdown_connections 2024-03-19T01:24:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.366+0000 7f93e500b640 1 -- 172.21.15.7:0/1582565978 wait complete. 2024-03-19T01:24:35.384 INFO:tasks.ceph:Scrubbing osd.4 2024-03-19T01:24:35.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-19T01:24:35.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.510+0000 7fc85383f640 1 Processor -- start 2024-03-19T01:24:35.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.510+0000 7fc85383f640 1 -- start start 2024-03-19T01:24:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.510+0000 7fc85383f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c148060 0x7fc84c1473f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.510+0000 7fc85383f640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fc84c06d1b0 0x7fc84c06dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.510+0000 7fc85383f640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84c06d610 con 0x7fc84c09ee00 2024-03-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.510+0000 7fc85383f640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fc84c06da50 con 0x7fc84c06d1b0 2024-03-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.510+0000 7fc85383f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc84c06bca0 con 0x7fc84c148060 2024-03-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc8515db640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c148060 0x7fc84c1473f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc850dda640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fc84c06d1b0 0x7fc84c06dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc8515db640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c148060 0x7fc84c1473f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:48318/0 (socket says 172.21.15.7:48318) 2024-03-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc851ddc640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fc84c09ee00 0x7fc84c0b56a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:59316/0 (socket says 172.21.15.7:59316) 2024-03-19T01:24:35.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc8515db640 1 -- 172.21.15.7:0/207537169 learned_addr learned my addr 172.21.15.7:0/207537169 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc850dda640 1 -- 172.21.15.7:0/207537169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c148060 msgr2=0x7fc84c1473f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc850dda640 1 --2- 172.21.15.7:0/207537169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c148060 0x7fc84c1473f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc850dda640 1 -- 172.21.15.7:0/207537169 >> v1:172.21.15.167:6789/0 conn(0x7fc84c09ee00 legacy=0x7fc84c0b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc850dda640 1 -- 172.21.15.7:0/207537169 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c06bf80 con 0x7fc84c06d1b0 2024-03-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc8515db640 1 --2- 172.21.15.7:0/207537169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c148060 0x7fc84c1473f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc850dda640 1 --2- 172.21.15.7:0/207537169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84c06d1b0 0x7fc84c06dee0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0097c0 tx=0x7fc83c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8adf7a1300135c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc843fff640 1 -- 172.21.15.7:0/207537169 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0b0070 con 0x7fc84c06d1b0 2024-03-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc843fff640 1 -- 172.21.15.7:0/207537169 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc83c0ab600 con 0x7fc84c06d1b0 2024-03-19T01:24:35.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc843fff640 1 -- 172.21.15.7:0/207537169 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0ab9d0 con 0x7fc84c06d1b0 2024-03-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc85383f640 1 -- 172.21.15.7:0/207537169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84c06d1b0 msgr2=0x7fc84c06dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.514+0000 7fc85383f640 1 --2- 172.21.15.7:0/207537169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84c06d1b0 0x7fc84c06dee0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0097c0 tx=0x7fc83c01c410 comp rx=0 tx=0).stop 2024-03-19T01:24:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- 172.21.15.7:0/207537169 shutdown_connections 2024-03-19T01:24:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 --2- 172.21.15.7:0/207537169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84c06d1b0 0x7fc84c06dee0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 --2- 172.21.15.7:0/207537169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c148060 0x7fc84c1473f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- 172.21.15.7:0/207537169 >> 172.21.15.7:0/207537169 conn(0x7fc84c0b6790 msgr2=0x7fc84c0b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- 172.21.15.7:0/207537169 shutdown_connections 2024-03-19T01:24:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- 172.21.15.7:0/207537169 wait complete. 2024-03-19T01:24:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 Processor -- start 2024-03-19T01:24:35.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- start start 2024-03-19T01:24:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84c06d1b0 0x7fc84c1466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c09ee00 0x7fc84c1e0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc84c148060 0x7fc84c1e0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c154e90 con 0x7fc84c148060 2024-03-19T01:24:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c154d10 con 0x7fc84c06d1b0 2024-03-19T01:24:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc84c155010 con 0x7fc84c09ee00 2024-03-19T01:24:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc851ddc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc84c148060 0x7fc84c1e0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc851ddc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc84c148060 0x7fc84c1e0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33780/0 (socket says 172.21.15.7:33780) 2024-03-19T01:24:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc851ddc640 1 -- 172.21.15.7:0/1848554171 learned_addr learned my addr 172.21.15.7:0/1848554171 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc850dda640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c09ee00 0x7fc84c1e0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc8515db640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84c06d1b0 0x7fc84c1466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc851ddc640 1 -- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c09ee00 msgr2=0x7fc84c1e0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc851ddc640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84c09ee00 0x7fc84c1e0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc851ddc640 1 -- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84c06d1b0 msgr2=0x7fc84c1466f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc851ddc640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84c06d1b0 0x7fc84c1466f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc851ddc640 1 -- 172.21.15.7:0/1848554171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c148fe0 con 0x7fc84c148060 2024-03-19T01:24:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc851ddc640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc84c148060 0x7fc84c1e0d50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc84400c730 tx=0x7fc84400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc841ffb640 1 -- 172.21.15.7:0/1848554171 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844018070 con 0x7fc84c148060 2024-03-19T01:24:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc841ffb640 1 -- 172.21.15.7:0/1848554171 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc844004d70 con 0x7fc84c148060 2024-03-19T01:24:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84c145260 con 0x7fc84c148060 2024-03-19T01:24:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc841ffb640 1 -- 172.21.15.7:0/1848554171 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84400a880 con 0x7fc84c148060 2024-03-19T01:24:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84c146e20 con 0x7fc84c148060 2024-03-19T01:24:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.518+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc84c075570 con 0x7fc84c148060 2024-03-19T01:24:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc841ffb640 1 -- 172.21.15.7:0/1848554171 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc84401b070 con 0x7fc84c148060 2024-03-19T01:24:35.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc841ffb640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc830067690 0x7fc830069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc841ffb640 1 -- 172.21.15.7:0/1848554171 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fc8440851d0 con 0x7fc84c148060 2024-03-19T01:24:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc8515db640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc830067690 0x7fc830069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc841ffb640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7fc83006d550 0x7fc83006f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc841ffb640 1 -- 172.21.15.7:0/1848554171 --> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc830070020 con 0x7fc83006d550 2024-03-19T01:24:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc841ffb640 1 -- 172.21.15.7:0/1848554171 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc844085580 con 0x7fc84c148060 2024-03-19T01:24:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc850dda640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7fc83006d550 0x7fc83006f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc8515db640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc830067690 0x7fc830069b50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc8480096d0 tx=0x7fc848009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.522+0000 7fc850dda640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7fc83006d550 0x7fc83006f970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.526+0000 7fc841ffb640 1 -- 172.21.15.7:0/1848554171 <== osd.4 v2:172.21.15.167:6800/2892054155 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc830070020 con 0x7fc83006d550 2024-03-19T01:24:35.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 --> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc84c070d30 con 0x7fc83006d550 2024-03-19T01:24:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc841ffb640 1 -- 172.21.15.7:0/1848554171 <== osd.4 v2:172.21.15.167:6800/2892054155 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc84c070d30 con 0x7fc83006d550 2024-03-19T01:24:35.542 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:35.542 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T01:24:35.542 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:35.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7fc83006d550 msgr2=0x7fc83006f970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.167:6800/2892054155,v1:172.21.15.167:6801/2892054155] conn(0x7fc83006d550 0x7fc83006f970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc830067690 msgr2=0x7fc830069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fc830067690 0x7fc830069b50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc8480096d0 tx=0x7fc848009240 comp rx=0 tx=0).stop 2024-03-19T01:24:35.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc84c148060 msgr2=0x7fc84c1e0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 --2- 172.21.15.7:0/1848554171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc84c148060 0x7fc84c1e0d50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc84400c730 tx=0x7fc84400cbf0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc851ddc640 1 -- 172.21.15.7:0/1848554171 reap_dead start 2024-03-19T01:24:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 shutdown_connections 2024-03-19T01:24:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 >> 172.21.15.7:0/1848554171 conn(0x7fc84c0b6790 msgr2=0x7fc84c070570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 shutdown_connections 2024-03-19T01:24:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.538+0000 7fc85383f640 1 -- 172.21.15.7:0/1848554171 wait complete. 2024-03-19T01:24:35.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-19T01:24:35.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f573e466640 1 Processor -- start 2024-03-19T01:24:35.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f573e466640 1 -- start start 2024-03-19T01:24:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f573e466640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f57380b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f573e466640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f57380ba740 0x7f5738148940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f573e466640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5738072a60 con 0x7f5738148e80 2024-03-19T01:24:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f573e466640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f57380b5560 con 0x7f57380ba740 2024-03-19T01:24:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f573e466640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f57380b5840 con 0x7f5738072680 2024-03-19T01:24:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f5737fff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f57380b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f57377fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f57380ba740 0x7f5738148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f5737fff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f57380b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:48344/0 (socket says 172.21.15.7:48344) 2024-03-19T01:24:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f573ca03640 1 --1- 172.21.15.7:0/135001242 >> v1:172.21.15.167:6789/0 conn(0x7f5738148e80 0x7f57380b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:59328/0 (socket says 172.21.15.7:59328) 2024-03-19T01:24:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f5737fff640 1 -- 172.21.15.7:0/135001242 learned_addr learned my addr 172.21.15.7:0/135001242 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f5737fff640 1 -- 172.21.15.7:0/135001242 >> v2:172.21.15.167:3300/0 conn(0x7f57380ba740 msgr2=0x7f5738148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f5737fff640 1 --2- 172.21.15.7:0/135001242 >> v2:172.21.15.167:3300/0 conn(0x7f57380ba740 0x7f5738148940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f5737fff640 1 -- 172.21.15.7:0/135001242 >> v1:172.21.15.167:6789/0 conn(0x7f5738148e80 legacy=0x7f57380b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f5737fff640 1 -- 172.21.15.7:0/135001242 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57380b4a00 con 0x7f5738072680 2024-03-19T01:24:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.666+0000 7f57377fe640 1 --2- 172.21.15.7:0/135001242 >> v2:172.21.15.167:3300/0 conn(0x7f57380ba740 0x7f5738148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:35.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f5737fff640 1 --2- 172.21.15.7:0/135001242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f57380b44c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5730009870 tx=0x7f573001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4c4582b59e2d7c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f5736ffd640 1 -- 172.21.15.7:0/135001242 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57300b1070 con 0x7f5738072680 2024-03-19T01:24:35.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f5736ffd640 1 -- 172.21.15.7:0/135001242 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57300ac620 con 0x7f5738072680 2024-03-19T01:24:35.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f5736ffd640 1 -- 172.21.15.7:0/135001242 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57300ac9f0 con 0x7f5738072680 2024-03-19T01:24:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 -- 172.21.15.7:0/135001242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 msgr2=0x7f57380b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 --2- 172.21.15.7:0/135001242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f57380b44c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5730009870 tx=0x7f573001c4b0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 -- 172.21.15.7:0/135001242 shutdown_connections 2024-03-19T01:24:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 --2- 172.21.15.7:0/135001242 >> v2:172.21.15.167:3300/0 conn(0x7f57380ba740 0x7f5738148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 --2- 172.21.15.7:0/135001242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f57380b44c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 -- 172.21.15.7:0/135001242 >> 172.21.15.7:0/135001242 conn(0x7f5738074130 msgr2=0x7f5738074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 -- 172.21.15.7:0/135001242 shutdown_connections 2024-03-19T01:24:35.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 -- 172.21.15.7:0/135001242 wait complete. 2024-03-19T01:24:35.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 Processor -- start 2024-03-19T01:24:35.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 -- start start 2024-03-19T01:24:35.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f573813f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57380ba740 0x7f573814a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5738148e80 0x7f5738142520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5738152820 con 0x7f57380ba740 2024-03-19T01:24:35.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f57381526a0 con 0x7f5738148e80 2024-03-19T01:24:35.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573e466640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f57381529a0 con 0x7f5738072680 2024-03-19T01:24:35.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f57377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57380ba740 0x7f573814a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f57377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57380ba740 0x7f573814a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33794/0 (socket says 172.21.15.7:33794) 2024-03-19T01:24:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f57377fe640 1 -- 172.21.15.7:0/863474043 learned_addr learned my addr 172.21.15.7:0/863474043 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f5737fff640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f573813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.670+0000 7f573ca03640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5738148e80 0x7f5738142520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f57377fe640 1 -- 172.21.15.7:0/863474043 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 msgr2=0x7f573813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f57377fe640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f573813f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f57377fe640 1 -- 172.21.15.7:0/863474043 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5738148e80 msgr2=0x7f5738142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f57377fe640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5738148e80 0x7f5738142520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f57377fe640 1 -- 172.21.15.7:0/863474043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f573814ae70 con 0x7f57380ba740 2024-03-19T01:24:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f57377fe640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57380ba740 0x7f573814a930 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f572000e930 tx=0x7f572000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f5734ff9640 1 -- 172.21.15.7:0/863474043 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720017070 con 0x7f57380ba740 2024-03-19T01:24:35.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f5734ff9640 1 -- 172.21.15.7:0/863474043 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f572000ca00 con 0x7f57380ba740 2024-03-19T01:24:35.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57381412c0 con 0x7f57380ba740 2024-03-19T01:24:35.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f5734ff9640 1 -- 172.21.15.7:0/863474043 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572000cdd0 con 0x7f57380ba740 2024-03-19T01:24:35.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f573814b3d0 con 0x7f57380ba740 2024-03-19T01:24:35.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5738075460 con 0x7f57380ba740 2024-03-19T01:24:35.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f5734ff9640 1 -- 172.21.15.7:0/863474043 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f572001a070 con 0x7f57380ba740 2024-03-19T01:24:35.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f5734ff9640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5718067660 0x7f5718069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f5734ff9640 1 -- 172.21.15.7:0/863474043 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f5720085390 con 0x7f57380ba740 2024-03-19T01:24:35.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f5737fff640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5718067660 0x7f5718069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.674+0000 7f5737fff640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5718067660 0x7f5718069b20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f57300ae000 tx=0x7f5730008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.678+0000 7f5734ff9640 1 -- 172.21.15.7:0/863474043 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f573814ae70 con 0x7f57380ba740 2024-03-19T01:24:35.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.830+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f5738149bc0 con 0x7f5718067660 2024-03-19T01:24:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f5734ff9640 1 -- 172.21.15.7:0/863474043 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5738149bc0 con 0x7f5718067660 2024-03-19T01:24:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 4 to deep-scrub 2024-03-19T01:24:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5718067660 msgr2=0x7f5718069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5718067660 0x7f5718069b20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f57300ae000 tx=0x7f5730008040 comp rx=0 tx=0).stop 2024-03-19T01:24:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57380ba740 msgr2=0x7f573814a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57380ba740 0x7f573814a930 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f572000e930 tx=0x7f572000edf0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 shutdown_connections 2024-03-19T01:24:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f5718067660 0x7f5718069b20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5738148e80 0x7f5738142520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57380ba740 0x7f573814a930 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 --2- 172.21.15.7:0/863474043 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5738072680 0x7f573813f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 >> 172.21.15.7:0/863474043 conn(0x7f5738074130 msgr2=0x7f5738067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 shutdown_connections 2024-03-19T01:24:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.834+0000 7f573e466640 1 -- 172.21.15.7:0/863474043 wait complete. 2024-03-19T01:24:35.855 INFO:tasks.ceph:Scrubbing osd.5 2024-03-19T01:24:35.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-19T01:24:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2693a05640 1 Processor -- start 2024-03-19T01:24:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2693a05640 1 -- start start 2024-03-19T01:24:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2693a05640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f268c06e050 0x7f268c09ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2693a05640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06ec30 0x7f268c06b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2693a05640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f268c14aec0 0x7f268c14b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2693a05640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f268c06e430 con 0x7f268c14aec0 2024-03-19T01:24:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2693a05640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f268c09f420 con 0x7f268c06e050 2024-03-19T01:24:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2693a05640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f268c072a00 con 0x7f268c06ec30 2024-03-19T01:24:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2691fa2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f268c14aec0 0x7f268c14b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2691fa2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f268c14aec0 0x7f268c14b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33802/0 (socket says 172.21.15.7:33802) 2024-03-19T01:24:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2691fa2640 1 -- 172.21.15.7:0/2649468014 learned_addr learned my addr 172.21.15.7:0/2649468014 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2690fa0640 1 --2- 172.21.15.7:0/2649468014 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06ec30 0x7f268c06b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f26917a1640 1 --2- 172.21.15.7:0/2649468014 >> v2:172.21.15.167:3300/0 conn(0x7f268c06e050 0x7f268c09ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2691fa2640 1 -- 172.21.15.7:0/2649468014 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06ec30 msgr2=0x7f268c06b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2691fa2640 1 --2- 172.21.15.7:0/2649468014 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06ec30 0x7f268c06b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2691fa2640 1 -- 172.21.15.7:0/2649468014 >> v2:172.21.15.167:3300/0 conn(0x7f268c06e050 msgr2=0x7f268c09ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2691fa2640 1 --2- 172.21.15.7:0/2649468014 >> v2:172.21.15.167:3300/0 conn(0x7f268c06e050 0x7f268c09ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.978+0000 7f2691fa2640 1 -- 172.21.15.7:0/2649468014 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c14b870 con 0x7f268c14aec0 2024-03-19T01:24:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2691fa2640 1 --2- 172.21.15.7:0/2649468014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 0x7f268c14b330 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f268c075090 tx=0x7f268401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17c13504bc4974 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2683fff640 1 -- 172.21.15.7:0/2649468014 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26840a55e0 con 0x7f268c14aec0 2024-03-19T01:24:35.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2683fff640 1 -- 172.21.15.7:0/2649468014 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26840a5780 con 0x7f268c14aec0 2024-03-19T01:24:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2683fff640 1 -- 172.21.15.7:0/2649468014 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26840a5b50 con 0x7f268c14aec0 2024-03-19T01:24:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 -- 172.21.15.7:0/2649468014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 msgr2=0x7f268c14b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 --2- 172.21.15.7:0/2649468014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 0x7f268c14b330 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f268c075090 tx=0x7f268401c960 comp rx=0 tx=0).stop 2024-03-19T01:24:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 -- 172.21.15.7:0/2649468014 shutdown_connections 2024-03-19T01:24:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 --2- 172.21.15.7:0/2649468014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 0x7f268c14b330 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 --2- 172.21.15.7:0/2649468014 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06ec30 0x7f268c06b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 --2- 172.21.15.7:0/2649468014 >> v2:172.21.15.167:3300/0 conn(0x7f268c06e050 0x7f268c09ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 -- 172.21.15.7:0/2649468014 >> 172.21.15.7:0/2649468014 conn(0x7f268c0b6790 msgr2=0x7f268c0b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 -- 172.21.15.7:0/2649468014 shutdown_connections 2024-03-19T01:24:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 -- 172.21.15.7:0/2649468014 wait complete. 2024-03-19T01:24:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 Processor -- start 2024-03-19T01:24:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.982+0000 7f2693a05640 1 -- start start 2024-03-19T01:24:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2693a05640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06e050 0x7f268c141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2693a05640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f268c06ec30 0x7f268c1e3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2693a05640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 0x7f268c1439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2693a05640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f268c0b6030 con 0x7f268c14aec0 2024-03-19T01:24:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2693a05640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f268c0b5eb0 con 0x7f268c06ec30 2024-03-19T01:24:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2693a05640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f268c0b61b0 con 0x7f268c06e050 2024-03-19T01:24:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2691fa2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 0x7f268c1439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2691fa2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 0x7f268c1439e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33810/0 (socket says 172.21.15.7:33810) 2024-03-19T01:24:35.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2691fa2640 1 -- 172.21.15.7:0/1224595845 learned_addr learned my addr 172.21.15.7:0/1224595845 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f26917a1640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06e050 0x7f268c141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2690fa0640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f268c06ec30 0x7f268c1e3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2691fa2640 1 -- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06e050 msgr2=0x7f268c141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2691fa2640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06e050 0x7f268c141960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2691fa2640 1 -- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f268c06ec30 msgr2=0x7f268c1e3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:35.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2691fa2640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f268c06ec30 0x7f268c1e3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2691fa2640 1 -- 172.21.15.7:0/1224595845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c1423f0 con 0x7f268c14aec0 2024-03-19T01:24:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f26917a1640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268c06e050 0x7f268c141960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2691fa2640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 0x7f268c1439e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f26840af040 tx=0x7f2684019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2681ffb640 1 -- 172.21.15.7:0/1224595845 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26840bc020 con 0x7f268c14aec0 2024-03-19T01:24:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2681ffb640 1 -- 172.21.15.7:0/1224595845 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26840b8690 con 0x7f268c14aec0 2024-03-19T01:24:35.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c13a1d0 con 0x7f268c14aec0 2024-03-19T01:24:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2681ffb640 1 -- 172.21.15.7:0/1224595845 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684009510 con 0x7f268c14aec0 2024-03-19T01:24:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c13ddb0 con 0x7f268c14aec0 2024-03-19T01:24:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f268c139db0 con 0x7f268c14aec0 2024-03-19T01:24:35.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2681ffb640 1 -- 172.21.15.7:0/1224595845 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f26840bf020 con 0x7f268c14aec0 2024-03-19T01:24:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2681ffb640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2670067660 0x7f2670069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f26917a1640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2670067660 0x7f2670069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2681ffb640 1 -- 172.21.15.7:0/1224595845 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f2684120930 con 0x7f268c14aec0 2024-03-19T01:24:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2681ffb640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7f267006d520 0x7f267006f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2681ffb640 1 -- 172.21.15.7:0/1224595845 --> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f267006fff0 con 0x7f267006d520 2024-03-19T01:24:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2681ffb640 1 -- 172.21.15.7:0/1224595845 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2684120dc0 con 0x7f268c14aec0 2024-03-19T01:24:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f26917a1640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2670067660 0x7f2670069b20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2688009840 tx=0x7f2688008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.986+0000 7f2690fa0640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7f267006d520 0x7f267006f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.990+0000 7f2690fa0640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7f267006d520 0x7f267006f940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:35.990+0000 7f2681ffb640 1 -- 172.21.15.7:0/1224595845 <== osd.5 v2:172.21.15.167:6824/3760007162 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f267006fff0 con 0x7f267006d520 2024-03-19T01:24:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 --> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f268c075570 con 0x7f267006d520 2024-03-19T01:24:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2681ffb640 1 -- 172.21.15.7:0/1224595845 <== osd.5 v2:172.21.15.167:6824/3760007162 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f268c075570 con 0x7f267006d520 2024-03-19T01:24:36.006 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:36.007 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T01:24:36.007 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7f267006d520 msgr2=0x7f267006f940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.167:6824/3760007162,v1:172.21.15.167:6825/3760007162] conn(0x7f267006d520 0x7f267006f940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2670067660 msgr2=0x7f2670069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f2670067660 0x7f2670069b20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2688009840 tx=0x7f2688008040 comp rx=0 tx=0).stop 2024-03-19T01:24:36.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 msgr2=0x7f268c1439e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 --2- 172.21.15.7:0/1224595845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f268c14aec0 0x7f268c1439e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f26840af040 tx=0x7f2684019700 comp rx=0 tx=0).stop 2024-03-19T01:24:36.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2691fa2640 1 -- 172.21.15.7:0/1224595845 reap_dead start 2024-03-19T01:24:36.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 shutdown_connections 2024-03-19T01:24:36.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 >> 172.21.15.7:0/1224595845 conn(0x7f268c0b6790 msgr2=0x7f268c0b4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:36.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 shutdown_connections 2024-03-19T01:24:36.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.002+0000 7f2693a05640 1 -- 172.21.15.7:0/1224595845 wait complete. 2024-03-19T01:24:36.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-19T01:24:36.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.126+0000 7fb38ced4640 1 Processor -- start 2024-03-19T01:24:36.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.126+0000 7fb38ced4640 1 -- start start 2024-03-19T01:24:36.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38ced4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb388144e10 0x7fb38814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38ced4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3880b1840 con 0x7fb3880b1460 2024-03-19T01:24:36.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38ced4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3880b2120 con 0x7fb38806d870 2024-03-19T01:24:36.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38ced4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb3881441e0 con 0x7fb388144e10 2024-03-19T01:24:36.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38659c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb3880b1460 0x7fb3880b6630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39604/0 (socket says 172.21.15.7:39604) 2024-03-19T01:24:36.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38659c640 1 -- 172.21.15.7:0/4108394347 learned_addr learned my addr 172.21.15.7:0/4108394347 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:36.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1835899757 0 0) 0x7fb3880b1840 con 0x7fb3880b1460 2024-03-19T01:24:36.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb386d9d640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb388144e10 0x7fb38814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb374003530 con 0x7fb3880b1460 2024-03-19T01:24:36.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1860000093 0 0) 0x7fb374003530 con 0x7fb3880b1460 2024-03-19T01:24:36.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb388144e10 msgr2=0x7fb38814f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:36.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb388144e10 0x7fb38814f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 >> v1:172.21.15.167:6789/0 conn(0x7fb38806d870 legacy=0x7fb388144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3880b85c0 con 0x7fb3880b1460 2024-03-19T01:24:36.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7fb380001680 con 0x7fb3880b1460 2024-03-19T01:24:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 >> v1:172.21.15.7:6789/0 conn(0x7fb3880b1460 legacy=0x7fb3880b6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3740039f0 0x7fb374003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb374004810 0x7fb374004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb388144e10 0x7fb374005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb3881441e0 con 0x7fb374004810 2024-03-19T01:24:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb37400bb30 con 0x7fb3740039f0 2024-03-19T01:24:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb37400df40 con 0x7fb388144e10 2024-03-19T01:24:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb385d9b640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb374004810 0x7fb374004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb38659c640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3740039f0 0x7fb374003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb385d9b640 1 -- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb388144e10 msgr2=0x7fb374005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb385d9b640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb388144e10 0x7fb374005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb385d9b640 1 -- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3740039f0 msgr2=0x7fb374003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb385d9b640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3740039f0 0x7fb374003e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb385d9b640 1 -- 172.21.15.7:0/4108394347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb374003530 con 0x7fb374004810 2024-03-19T01:24:36.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.130+0000 7fb385d9b640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb374004810 0x7fb374004c90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb370001780 tx=0x7fb37001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b5b4edb01e24b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3700013a0 con 0x7fb374004810 2024-03-19T01:24:36.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3700052f0 con 0x7fb374004810 2024-03-19T01:24:36.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38559a640 1 -- 172.21.15.7:0/4108394347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370003840 con 0x7fb374004810 2024-03-19T01:24:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- 172.21.15.7:0/4108394347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb374004810 msgr2=0x7fb374004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 --2- 172.21.15.7:0/4108394347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb374004810 0x7fb374004c90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb370001780 tx=0x7fb37001c420 comp rx=0 tx=0).stop 2024-03-19T01:24:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 -- 172.21.15.7:0/4108394347 reap_dead start 2024-03-19T01:24:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- 172.21.15.7:0/4108394347 shutdown_connections 2024-03-19T01:24:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- 172.21.15.7:0/4108394347 >> 172.21.15.7:0/4108394347 conn(0x7fb3880b5330 msgr2=0x7fb3880b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- 172.21.15.7:0/4108394347 shutdown_connections 2024-03-19T01:24:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- 172.21.15.7:0/4108394347 wait complete. 2024-03-19T01:24:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 Processor -- start 2024-03-19T01:24:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- start start 2024-03-19T01:24:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb374004810 0x7fb3881d5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3881e06d0 0x7fb3881d6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3881e2f50 0x7fb3881e3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb3880b7b80 con 0x7fb3881e2f50 2024-03-19T01:24:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb3880b7a00 con 0x7fb374004810 2024-03-19T01:24:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb3880b7d00 con 0x7fb3881e06d0 2024-03-19T01:24:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3881e2f50 0x7fb3881e3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3881e2f50 0x7fb3881e3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33828/0 (socket says 172.21.15.7:33828) 2024-03-19T01:24:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 -- 172.21.15.7:0/433629082 learned_addr learned my addr 172.21.15.7:0/433629082 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38659c640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb374004810 0x7fb3881d5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb385d9b640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3881e06d0 0x7fb3881d6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 -- 172.21.15.7:0/433629082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3881e06d0 msgr2=0x7fb3881d6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3881e06d0 0x7fb3881d6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 -- 172.21.15.7:0/433629082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb374004810 msgr2=0x7fb3881d5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb374004810 0x7fb3881d5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 -- 172.21.15.7:0/433629082 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38813ec70 con 0x7fb3881e2f50 2024-03-19T01:24:36.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb386d9d640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3881e2f50 0x7fb3881e3400 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb374004790 tx=0x7fb37800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb37effd640 1 -- 172.21.15.7:0/433629082 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378018020 con 0x7fb3881e2f50 2024-03-19T01:24:36.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb37effd640 1 -- 172.21.15.7:0/433629082 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb37800bc80 con 0x7fb3881e2f50 2024-03-19T01:24:36.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb37effd640 1 -- 172.21.15.7:0/433629082 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378014640 con 0x7fb3881e2f50 2024-03-19T01:24:36.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3881dfbf0 con 0x7fb3881e2f50 2024-03-19T01:24:36.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.134+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3881e01b0 con 0x7fb3881e2f50 2024-03-19T01:24:36.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.138+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3881df9e0 con 0x7fb3881e2f50 2024-03-19T01:24:36.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.138+0000 7fb37effd640 1 -- 172.21.15.7:0/433629082 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb37801b020 con 0x7fb3881e2f50 2024-03-19T01:24:36.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.138+0000 7fb37effd640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb368067690 0x7fb368069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.138+0000 7fb37effd640 1 -- 172.21.15.7:0/433629082 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fb378085c50 con 0x7fb3881e2f50 2024-03-19T01:24:36.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.138+0000 7fb38659c640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb368067690 0x7fb368069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.138+0000 7fb38659c640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb368067690 0x7fb368069b50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb380005a60 tx=0x7fb380001560 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.142+0000 7fb37effd640 1 -- 172.21.15.7:0/433629082 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb37808b020 con 0x7fb3881e2f50 2024-03-19T01:24:36.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fb388075460 con 0x7fb368067690 2024-03-19T01:24:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb37effd640 1 -- 172.21.15.7:0/433629082 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb388075460 con 0x7fb368067690 2024-03-19T01:24:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 5 to deep-scrub 2024-03-19T01:24:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb368067690 msgr2=0x7fb368069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb368067690 0x7fb368069b50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb380005a60 tx=0x7fb380001560 comp rx=0 tx=0).stop 2024-03-19T01:24:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3881e2f50 msgr2=0x7fb3881e3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3881e2f50 0x7fb3881e3400 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb374004790 tx=0x7fb37800b480 comp rx=0 tx=0).stop 2024-03-19T01:24:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 shutdown_connections 2024-03-19T01:24:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fb368067690 0x7fb368069b50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3881e06d0 0x7fb3881d6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3881e2f50 0x7fb3881e3400 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 --2- 172.21.15.7:0/433629082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb374004810 0x7fb3881d5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.302+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 >> 172.21.15.7:0/433629082 conn(0x7fb3880b5330 msgr2=0x7fb388143910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.306+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 shutdown_connections 2024-03-19T01:24:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.306+0000 7fb38ced4640 1 -- 172.21.15.7:0/433629082 wait complete. 2024-03-19T01:24:36.324 INFO:tasks.ceph:Scrubbing osd.6 2024-03-19T01:24:36.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-19T01:24:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e33d6b640 1 Processor -- start 2024-03-19T01:24:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e33d6b640 1 -- start start 2024-03-19T01:24:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e33d6b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e2c0b7790 0x7f6e2c14a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e33d6b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f6e2c06da80 0x7f6e2c141410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e33d6b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2c0b5a20 con 0x7f6e2c141950 2024-03-19T01:24:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e33d6b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7f6e2c06d280 con 0x7f6e2c0b7790 2024-03-19T01:24:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e33d6b640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f6e2c14aeb0 con 0x7f6e2c06da80 2024-03-19T01:24:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e31b07640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e2c0b7790 0x7f6e2c14a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e32308640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6e2c141950 0x7f6e2c0b6160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39614/0 (socket says 172.21.15.7:39614) 2024-03-19T01:24:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e32308640 1 -- 172.21.15.7:0/947952847 learned_addr learned my addr 172.21.15.7:0/947952847 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e31306640 1 --2- 172.21.15.7:0/947952847 >> v2:172.21.15.167:3300/0 conn(0x7f6e2c06da80 0x7f6e2c141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e31b07640 1 -- 172.21.15.7:0/947952847 >> v2:172.21.15.167:3300/0 conn(0x7f6e2c06da80 msgr2=0x7f6e2c141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e31b07640 1 --2- 172.21.15.7:0/947952847 >> v2:172.21.15.167:3300/0 conn(0x7f6e2c06da80 0x7f6e2c141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e31b07640 1 -- 172.21.15.7:0/947952847 >> v1:172.21.15.7:6789/0 conn(0x7f6e2c141950 legacy=0x7f6e2c0b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e31b07640 1 -- 172.21.15.7:0/947952847 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c0b7b70 con 0x7f6e2c0b7790 2024-03-19T01:24:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.442+0000 7f6e30b05640 1 -- 172.21.15.7:0/947952847 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3755483483 0 0) 0x7f6e2c0b5a20 con 0x7f6e2c141950 2024-03-19T01:24:36.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e31b07640 1 --2- 172.21.15.7:0/947952847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c0b7790 0x7f6e2c14a970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c0b5300 tx=0x7f6e2801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd7610542ea477d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e30b05640 1 -- 172.21.15.7:0/947952847 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e280b0070 con 0x7f6e2c0b7790 2024-03-19T01:24:36.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e30b05640 1 -- 172.21.15.7:0/947952847 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e280ab600 con 0x7f6e2c0b7790 2024-03-19T01:24:36.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e30b05640 1 -- 172.21.15.7:0/947952847 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e280ab9d0 con 0x7f6e2c0b7790 2024-03-19T01:24:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 -- 172.21.15.7:0/947952847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c0b7790 msgr2=0x7f6e2c14a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 --2- 172.21.15.7:0/947952847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c0b7790 0x7f6e2c14a970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c0b5300 tx=0x7f6e2801c360 comp rx=0 tx=0).stop 2024-03-19T01:24:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 -- 172.21.15.7:0/947952847 shutdown_connections 2024-03-19T01:24:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 --2- 172.21.15.7:0/947952847 >> v2:172.21.15.167:3300/0 conn(0x7f6e2c06da80 0x7f6e2c141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 --2- 172.21.15.7:0/947952847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c0b7790 0x7f6e2c14a970 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 -- 172.21.15.7:0/947952847 >> 172.21.15.7:0/947952847 conn(0x7f6e2c06b2c0 msgr2=0x7f6e2c06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 -- 172.21.15.7:0/947952847 shutdown_connections 2024-03-19T01:24:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 -- 172.21.15.7:0/947952847 wait complete. 2024-03-19T01:24:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 Processor -- start 2024-03-19T01:24:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.446+0000 7f6e33d6b640 1 -- start start 2024-03-19T01:24:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e33d6b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e2c06da80 0x7f6e2c1441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e33d6b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e2c0b7790 0x7f6e2c1423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e33d6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c141950 0x7f6e2c143270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e33d6b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2c154ef0 con 0x7f6e2c141950 2024-03-19T01:24:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e33d6b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2c06e460 con 0x7f6e2c0b7790 2024-03-19T01:24:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e33d6b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6e2c155070 con 0x7f6e2c06da80 2024-03-19T01:24:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e32308640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c141950 0x7f6e2c143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e32308640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c141950 0x7f6e2c143270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33848/0 (socket says 172.21.15.7:33848) 2024-03-19T01:24:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e32308640 1 -- 172.21.15.7:0/1824291883 learned_addr learned my addr 172.21.15.7:0/1824291883 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e31306640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e2c0b7790 0x7f6e2c1423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e31b07640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e2c06da80 0x7f6e2c1441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e32308640 1 -- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e2c06da80 msgr2=0x7f6e2c1441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e32308640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e2c06da80 0x7f6e2c1441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e32308640 1 -- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e2c0b7790 msgr2=0x7f6e2c1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e32308640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e2c0b7790 0x7f6e2c1423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e32308640 1 -- 172.21.15.7:0/1824291883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c1446e0 con 0x7f6e2c141950 2024-03-19T01:24:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e31306640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e2c0b7790 0x7f6e2c1423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e31b07640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e2c06da80 0x7f6e2c1441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e32308640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c141950 0x7f6e2c143270 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400c750 tx=0x7f6e2400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e1e7fc640 1 -- 172.21.15.7:0/1824291883 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e24018070 con 0x7f6e2c141950 2024-03-19T01:24:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e1e7fc640 1 -- 172.21.15.7:0/1824291883 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e2400a640 con 0x7f6e2c141950 2024-03-19T01:24:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2c141ee0 con 0x7f6e2c141950 2024-03-19T01:24:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e1e7fc640 1 -- 172.21.15.7:0/1824291883 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2400ce60 con 0x7f6e2c141950 2024-03-19T01:24:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e2c143b50 con 0x7f6e2c141950 2024-03-19T01:24:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6e2c075570 con 0x7f6e2c141950 2024-03-19T01:24:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e1e7fc640 1 -- 172.21.15.7:0/1824291883 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6e2401b070 con 0x7f6e2c141950 2024-03-19T01:24:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e1e7fc640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6e10067660 0x7f6e10069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e1e7fc640 1 -- 172.21.15.7:0/1824291883 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f6e24085130 con 0x7f6e2c141950 2024-03-19T01:24:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e31b07640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6e10067660 0x7f6e10069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e1e7fc640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f6e1006d520 0x7f6e1006f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e1e7fc640 1 -- 172.21.15.7:0/1824291883 --> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e2401fe50 con 0x7f6e1006d520 2024-03-19T01:24:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e1e7fc640 1 -- 172.21.15.7:0/1824291883 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6e240855c0 con 0x7f6e2c141950 2024-03-19T01:24:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e31b07640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6e10067660 0x7f6e10069b20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6e280ad000 tx=0x7f6e28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.450+0000 7f6e31306640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f6e1006d520 0x7f6e1006f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.454+0000 7f6e31306640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f6e1006d520 0x7f6e1006f940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.454+0000 7f6e1e7fc640 1 -- 172.21.15.7:0/1824291883 <== osd.6 v2:172.21.15.167:6803/2582461346 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6e2401fe50 con 0x7f6e1006d520 2024-03-19T01:24:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.466+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 --> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6e2c146c80 con 0x7f6e1006d520 2024-03-19T01:24:36.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e1e7fc640 1 -- 172.21.15.7:0/1824291883 <== osd.6 v2:172.21.15.167:6803/2582461346 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6e2c146c80 con 0x7f6e1006d520 2024-03-19T01:24:36.471 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T01:24:36.472 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:36.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f6e1006d520 msgr2=0x7f6e1006f940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.167:6803/2582461346,v1:172.21.15.167:6806/2582461346] conn(0x7f6e1006d520 0x7f6e1006f940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6e10067660 msgr2=0x7f6e10069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f6e10067660 0x7f6e10069b20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6e280ad000 tx=0x7f6e28008040 comp rx=0 tx=0).stop 2024-03-19T01:24:36.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c141950 msgr2=0x7f6e2c143270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 --2- 172.21.15.7:0/1824291883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e2c141950 0x7f6e2c143270 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400c750 tx=0x7f6e2400cc10 comp rx=0 tx=0).stop 2024-03-19T01:24:36.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e32308640 1 -- 172.21.15.7:0/1824291883 reap_dead start 2024-03-19T01:24:36.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 shutdown_connections 2024-03-19T01:24:36.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 >> 172.21.15.7:0/1824291883 conn(0x7f6e2c06b2c0 msgr2=0x7f6e2c06b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:36.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 shutdown_connections 2024-03-19T01:24:36.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.470+0000 7f6e33d6b640 1 -- 172.21.15.7:0/1824291883 wait complete. 2024-03-19T01:24:36.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-19T01:24:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f6e36640 1 Processor -- start 2024-03-19T01:24:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f6e36640 1 -- start start 2024-03-19T01:24:36.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f6e36640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f00bd660 0x7fe2f014a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f6e36640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2f014ae60 0x7fe2f0151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f6e36640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f00b6ce0 con 0x7fe2f00b4570 2024-03-19T01:24:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f6e36640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f00b4950 con 0x7fe2f014ae60 2024-03-19T01:24:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f6e36640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f00b51c0 con 0x7fe2f00bd660 2024-03-19T01:24:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f53d3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2f014ae60 0x7fe2f0151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f53d3640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2f014ae60 0x7fe2f0151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33852/0 (socket says 172.21.15.7:33852) 2024-03-19T01:24:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f53d3640 1 -- 172.21.15.7:0/2417394972 learned_addr learned my addr 172.21.15.7:0/2417394972 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2effff640 1 --2- 172.21.15.7:0/2417394972 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f00bd660 0x7fe2f014a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f53d3640 1 -- 172.21.15.7:0/2417394972 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f00bd660 msgr2=0x7fe2f014a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f53d3640 1 --2- 172.21.15.7:0/2417394972 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f00bd660 0x7fe2f014a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f53d3640 1 -- 172.21.15.7:0/2417394972 >> v1:172.21.15.167:6789/0 conn(0x7fe2f00b4570 legacy=0x7fe2f0140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.598+0000 7fe2f53d3640 1 -- 172.21.15.7:0/2417394972 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f00b5480 con 0x7fe2f014ae60 2024-03-19T01:24:36.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 --2- 172.21.15.7:0/2417394972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f014ae60 0x7fe2f0151cb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009e50 tx=0x7fe2e801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f6e9507ac65e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2ef7fe640 1 -- 172.21.15.7:0/2417394972 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e80a55d0 con 0x7fe2f014ae60 2024-03-19T01:24:36.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2ef7fe640 1 -- 172.21.15.7:0/2417394972 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e80a5770 con 0x7fe2f014ae60 2024-03-19T01:24:36.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2ef7fe640 1 -- 172.21.15.7:0/2417394972 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e80a5b40 con 0x7fe2f014ae60 2024-03-19T01:24:36.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- 172.21.15.7:0/2417394972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f014ae60 msgr2=0x7fe2f0151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 --2- 172.21.15.7:0/2417394972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f014ae60 0x7fe2f0151cb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009e50 tx=0x7fe2e801c910 comp rx=0 tx=0).stop 2024-03-19T01:24:36.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- 172.21.15.7:0/2417394972 shutdown_connections 2024-03-19T01:24:36.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 --2- 172.21.15.7:0/2417394972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f014ae60 0x7fe2f0151cb0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 --2- 172.21.15.7:0/2417394972 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f00bd660 0x7fe2f014a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- 172.21.15.7:0/2417394972 >> 172.21.15.7:0/2417394972 conn(0x7fe2f0072df0 msgr2=0x7fe2f0073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:36.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- 172.21.15.7:0/2417394972 shutdown_connections 2024-03-19T01:24:36.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- 172.21.15.7:0/2417394972 wait complete. 2024-03-19T01:24:36.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 Processor -- start 2024-03-19T01:24:36.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- start start 2024-03-19T01:24:36.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2f00b4570 0x7fe2f0139520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f014ae60 0x7fe2f0149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f0148fc0 0x7fe2f0149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f00b93b0 con 0x7fe2f0148fc0 2024-03-19T01:24:36.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f00b9230 con 0x7fe2f00b4570 2024-03-19T01:24:36.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f0154a10 con 0x7fe2f014ae60 2024-03-19T01:24:36.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f0148fc0 0x7fe2f0149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f0148fc0 0x7fe2f0149450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33854/0 (socket says 172.21.15.7:33854) 2024-03-19T01:24:36.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 -- 172.21.15.7:0/264160857 learned_addr learned my addr 172.21.15.7:0/264160857 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:36.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f4bd2640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2f00b4570 0x7fe2f0139520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2effff640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f014ae60 0x7fe2f0149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 -- 172.21.15.7:0/264160857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f014ae60 msgr2=0x7fe2f0149a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f014ae60 0x7fe2f0149a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 -- 172.21.15.7:0/264160857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2f00b4570 msgr2=0x7fe2f0139520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2f00b4570 0x7fe2f0139520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 -- 172.21.15.7:0/264160857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f014a5c0 con 0x7fe2f0148fc0 2024-03-19T01:24:36.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f53d3640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f0148fc0 0x7fe2f0149450 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe2e801a540 tx=0x7fe2e80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2ed7fa640 1 -- 172.21.15.7:0/264160857 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e80bc070 con 0x7fe2f0148fc0 2024-03-19T01:24:36.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2ed7fa640 1 -- 172.21.15.7:0/264160857 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e80b8660 con 0x7fe2f0148fc0 2024-03-19T01:24:36.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f013e590 con 0x7fe2f0148fc0 2024-03-19T01:24:36.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.602+0000 7fe2ed7fa640 1 -- 172.21.15.7:0/264160857 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8009510 con 0x7fe2f0148fc0 2024-03-19T01:24:36.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.606+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f0148590 con 0x7fe2f0148fc0 2024-03-19T01:24:36.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.606+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f0075460 con 0x7fe2f0148fc0 2024-03-19T01:24:36.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.606+0000 7fe2ed7fa640 1 -- 172.21.15.7:0/264160857 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe2e80bf070 con 0x7fe2f0148fc0 2024-03-19T01:24:36.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.606+0000 7fe2ed7fa640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe2d0067660 0x7fe2d0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.606+0000 7fe2f4bd2640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe2d0067660 0x7fe2d0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.606+0000 7fe2ed7fa640 1 -- 172.21.15.7:0/264160857 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fe2e81204c0 con 0x7fe2f0148fc0 2024-03-19T01:24:36.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.606+0000 7fe2f4bd2640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe2d0067660 0x7fe2d0069b20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8001650 tx=0x7fe2d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.610+0000 7fe2ed7fa640 1 -- 172.21.15.7:0/264160857 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe2e8016620 con 0x7fe2f0148fc0 2024-03-19T01:24:36.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fe2f00733e0 con 0x7fe2d0067660 2024-03-19T01:24:36.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2ed7fa640 1 -- 172.21.15.7:0/264160857 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe2f00733e0 con 0x7fe2d0067660 2024-03-19T01:24:36.777 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 6 to deep-scrub 2024-03-19T01:24:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe2d0067660 msgr2=0x7fe2d0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe2d0067660 0x7fe2d0069b20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8001650 tx=0x7fe2d8009040 comp rx=0 tx=0).stop 2024-03-19T01:24:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f0148fc0 msgr2=0x7fe2f0149450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f0148fc0 0x7fe2f0149450 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe2e801a540 tx=0x7fe2e80196b0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 shutdown_connections 2024-03-19T01:24:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe2d0067660 0x7fe2d0069b20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2f014ae60 0x7fe2f0149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f0148fc0 0x7fe2f0149450 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 --2- 172.21.15.7:0/264160857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2f00b4570 0x7fe2f0139520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 >> 172.21.15.7:0/264160857 conn(0x7fe2f0072df0 msgr2=0x7fe2f0140440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:36.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 shutdown_connections 2024-03-19T01:24:36.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.774+0000 7fe2f6e36640 1 -- 172.21.15.7:0/264160857 wait complete. 2024-03-19T01:24:36.796 INFO:tasks.ceph:Scrubbing osd.7 2024-03-19T01:24:36.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-19T01:24:36.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0c5572640 1 Processor -- start 2024-03-19T01:24:36.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0c5572640 1 -- start start 2024-03-19T01:24:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0c5572640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe0c006e3c0 0x7fe0c0072190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0c5572640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0c0071930 0x7fe0c0141380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0c5572640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fe0c01418c0 0x7fe0c01443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0c5572640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7fe0c0071240 con 0x7fe0c006e3c0 2024-03-19T01:24:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0c5572640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fe0c0070c30 con 0x7fe0c01418c0 2024-03-19T01:24:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0c5572640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c006e7a0 con 0x7fe0c0071930 2024-03-19T01:24:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0beffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe0c006e3c0 0x7fe0c0072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0beffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe0c006e3c0 0x7fe0c0072190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33864/0 (socket says 172.21.15.7:33864) 2024-03-19T01:24:36.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0beffd640 1 -- 172.21.15.7:0/4283495962 learned_addr learned my addr 172.21.15.7:0/4283495962 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:36.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0b7fff640 1 --2- 172.21.15.7:0/4283495962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0c0071930 0x7fe0c0141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0bf7fe640 1 --2- 172.21.15.7:0/4283495962 >> v2:172.21.15.167:3300/0 conn(0x7fe0c01418c0 0x7fe0c01443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0beffd640 1 -- 172.21.15.7:0/4283495962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0c0071930 msgr2=0x7fe0c0141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:36.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0beffd640 1 --2- 172.21.15.7:0/4283495962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0c0071930 0x7fe0c0141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0beffd640 1 -- 172.21.15.7:0/4283495962 >> v2:172.21.15.167:3300/0 conn(0x7fe0c01418c0 msgr2=0x7fe0c01443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:36.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0beffd640 1 --2- 172.21.15.7:0/4283495962 >> v2:172.21.15.167:3300/0 conn(0x7fe0c01418c0 0x7fe0c01443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0beffd640 1 -- 172.21.15.7:0/4283495962 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c00728a0 con 0x7fe0c006e3c0 2024-03-19T01:24:36.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.922+0000 7fe0bf7fe640 1 --2- 172.21.15.7:0/4283495962 >> v2:172.21.15.167:3300/0 conn(0x7fe0c01418c0 0x7fe0c01443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:36.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0beffd640 1 --2- 172.21.15.7:0/4283495962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c006e3c0 0x7fe0c0072190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00b5f30 tx=0x7fe0b001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7c4dee43d0f5db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0be7fc640 1 -- 172.21.15.7:0/4283495962 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b00b0070 con 0x7fe0c006e3c0 2024-03-19T01:24:36.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0be7fc640 1 -- 172.21.15.7:0/4283495962 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b00ab600 con 0x7fe0c006e3c0 2024-03-19T01:24:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0be7fc640 1 -- 172.21.15.7:0/4283495962 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b00ab9d0 con 0x7fe0c006e3c0 2024-03-19T01:24:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- 172.21.15.7:0/4283495962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c006e3c0 msgr2=0x7fe0c0072190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 --2- 172.21.15.7:0/4283495962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c006e3c0 0x7fe0c0072190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00b5f30 tx=0x7fe0b001c400 comp rx=0 tx=0).stop 2024-03-19T01:24:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- 172.21.15.7:0/4283495962 shutdown_connections 2024-03-19T01:24:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 --2- 172.21.15.7:0/4283495962 >> v2:172.21.15.167:3300/0 conn(0x7fe0c01418c0 0x7fe0c01443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 --2- 172.21.15.7:0/4283495962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0c0071930 0x7fe0c0141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 --2- 172.21.15.7:0/4283495962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c006e3c0 0x7fe0c0072190 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- 172.21.15.7:0/4283495962 >> 172.21.15.7:0/4283495962 conn(0x7fe0c00b6610 msgr2=0x7fe0c00b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- 172.21.15.7:0/4283495962 shutdown_connections 2024-03-19T01:24:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- 172.21.15.7:0/4283495962 wait complete. 2024-03-19T01:24:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 Processor -- start 2024-03-19T01:24:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- start start 2024-03-19T01:24:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0c006e3c0 0x7fe0c00b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c01418c0 0x7fe0c014dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0c0145700 0x7fe0c006b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c01481f0 con 0x7fe0c01418c0 2024-03-19T01:24:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c0148070 con 0x7fe0c006e3c0 2024-03-19T01:24:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c0148370 con 0x7fe0c0145700 2024-03-19T01:24:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c01418c0 0x7fe0c014dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c01418c0 0x7fe0c014dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33878/0 (socket says 172.21.15.7:33878) 2024-03-19T01:24:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b7fff640 1 -- 172.21.15.7:0/855661966 learned_addr learned my addr 172.21.15.7:0/855661966 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:36.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0beffd640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0c006e3c0 0x7fe0c00b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0bf7fe640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0c0145700 0x7fe0c006b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b7fff640 1 -- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0c0145700 msgr2=0x7fe0c006b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b7fff640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0c0145700 0x7fe0c006b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b7fff640 1 -- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0c006e3c0 msgr2=0x7fe0c00b8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b7fff640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0c006e3c0 0x7fe0c00b8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b7fff640 1 -- 172.21.15.7:0/855661966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c0146bd0 con 0x7fe0c01418c0 2024-03-19T01:24:36.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b7fff640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c01418c0 0x7fe0c014dc40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800d800 tx=0x7fe0a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b77fe640 1 -- 172.21.15.7:0/855661966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8018020 con 0x7fe0c01418c0 2024-03-19T01:24:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b77fe640 1 -- 172.21.15.7:0/855661966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a80096a0 con 0x7fe0c01418c0 2024-03-19T01:24:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c0145bc0 con 0x7fe0c01418c0 2024-03-19T01:24:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0b77fe640 1 -- 172.21.15.7:0/855661966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8009a50 con 0x7fe0c01418c0 2024-03-19T01:24:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.926+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c014e180 con 0x7fe0c01418c0 2024-03-19T01:24:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe0c0075570 con 0x7fe0c01418c0 2024-03-19T01:24:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0b77fe640 1 -- 172.21.15.7:0/855661966 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe0a801b020 con 0x7fe0c01418c0 2024-03-19T01:24:36.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0b77fe640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe0a0067660 0x7fe0a0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0b77fe640 1 -- 172.21.15.7:0/855661966 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fe0a8085b10 con 0x7fe0c01418c0 2024-03-19T01:24:36.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0beffd640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe0a0067660 0x7fe0a0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0b77fe640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7fe0a006d520 0x7fe0a006f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:36.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0b77fe640 1 -- 172.21.15.7:0/855661966 --> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0a800af30 con 0x7fe0a006d520 2024-03-19T01:24:36.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0b77fe640 1 -- 172.21.15.7:0/855661966 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe0a808a2c0 con 0x7fe0c01418c0 2024-03-19T01:24:36.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0beffd640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe0a0067660 0x7fe0a0069b20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe0b00ad000 tx=0x7fe0b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0bf7fe640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7fe0a006d520 0x7fe0a006f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:36.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.930+0000 7fe0bf7fe640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7fe0a006d520 0x7fe0a006f940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:36.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.934+0000 7fe0b77fe640 1 -- 172.21.15.7:0/855661966 <== osd.7 v2:172.21.15.167:6802/2393181242 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0a800af30 con 0x7fe0a006d520 2024-03-19T01:24:36.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 --> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe0c0142d30 con 0x7fe0a006d520 2024-03-19T01:24:36.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0b77fe640 1 -- 172.21.15.7:0/855661966 <== osd.7 v2:172.21.15.167:6802/2393181242 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe0c0142d30 con 0x7fe0a006d520 2024-03-19T01:24:36.950 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-19T01:24:36.950 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T01:24:36.950 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-19T01:24:36.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7fe0a006d520 msgr2=0x7fe0a006f940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.167:6802/2393181242,v1:172.21.15.167:6805/2393181242] conn(0x7fe0a006d520 0x7fe0a006f940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe0a0067660 msgr2=0x7fe0a0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fe0a0067660 0x7fe0a0069b20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe0b00ad000 tx=0x7fe0b0008040 comp rx=0 tx=0).stop 2024-03-19T01:24:36.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c01418c0 msgr2=0x7fe0c014dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:36.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 --2- 172.21.15.7:0/855661966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0c01418c0 0x7fe0c014dc40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800d800 tx=0x7fe0a800dcc0 comp rx=0 tx=0).stop 2024-03-19T01:24:36.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0bf7fe640 1 -- 172.21.15.7:0/855661966 reap_dead start 2024-03-19T01:24:36.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 shutdown_connections 2024-03-19T01:24:36.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 >> 172.21.15.7:0/855661966 conn(0x7fe0c00b6610 msgr2=0x7fe0c006d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:36.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 shutdown_connections 2024-03-19T01:24:36.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:36.946+0000 7fe0c5572640 1 -- 172.21.15.7:0/855661966 wait complete. 2024-03-19T01:24:36.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-19T01:24:37.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.066+0000 7fd415cf3640 1 Processor -- start 2024-03-19T01:24:37.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.066+0000 7fd415cf3640 1 -- start start 2024-03-19T01:24:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd415cf3640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fd4100b1460 0x7fd4100b6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd415cf3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 0x7fd410144700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd415cf3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4100b1840 con 0x7fd4100b80f0 2024-03-19T01:24:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd415cf3640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fd4100b2120 con 0x7fd4100b1460 2024-03-19T01:24:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd415cf3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd4101441e0 con 0x7fd41006d870 2024-03-19T01:24:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40effd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 0x7fd410144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40f7fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fd4100b1460 0x7fd4100b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40f7fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fd4100b1460 0x7fd4100b6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.7:38668/0 (socket says 172.21.15.7:38668) 2024-03-19T01:24:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40f7fe640 1 -- 172.21.15.7:0/4196452293 learned_addr learned my addr 172.21.15.7:0/4196452293 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40ffff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fd4100b80f0 0x7fd410147640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:59342/0 (socket says 172.21.15.7:59342) 2024-03-19T01:24:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40f7fe640 1 -- 172.21.15.7:0/4196452293 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 msgr2=0x7fd410144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40f7fe640 1 --2- 172.21.15.7:0/4196452293 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 0x7fd410144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40f7fe640 1 -- 172.21.15.7:0/4196452293 >> v1:172.21.15.167:6789/0 conn(0x7fd4100b80f0 legacy=0x7fd410147640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40f7fe640 1 -- 172.21.15.7:0/4196452293 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4100b85c0 con 0x7fd4100b1460 2024-03-19T01:24:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.070+0000 7fd40f7fe640 1 --2- 172.21.15.7:0/4196452293 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 0x7fd4100b6630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd40000bb10 tx=0x7fd40001eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=876c0d8cf02d6836 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40e7fc640 1 -- 172.21.15.7:0/4196452293 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4000b4070 con 0x7fd4100b1460 2024-03-19T01:24:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40e7fc640 1 -- 172.21.15.7:0/4196452293 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4000a7a70 con 0x7fd4100b1460 2024-03-19T01:24:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40e7fc640 1 -- 172.21.15.7:0/4196452293 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4000a7e40 con 0x7fd4100b1460 2024-03-19T01:24:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- 172.21.15.7:0/4196452293 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 msgr2=0x7fd4100b6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 --2- 172.21.15.7:0/4196452293 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 0x7fd4100b6630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd40000bb10 tx=0x7fd40001eba0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- 172.21.15.7:0/4196452293 shutdown_connections 2024-03-19T01:24:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 --2- 172.21.15.7:0/4196452293 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 0x7fd410144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 --2- 172.21.15.7:0/4196452293 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 0x7fd4100b6630 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- 172.21.15.7:0/4196452293 >> 172.21.15.7:0/4196452293 conn(0x7fd4100b5330 msgr2=0x7fd4100b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- 172.21.15.7:0/4196452293 shutdown_connections 2024-03-19T01:24:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- 172.21.15.7:0/4196452293 wait complete. 2024-03-19T01:24:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 Processor -- start 2024-03-19T01:24:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- start start 2024-03-19T01:24:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 0x7fd41013ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 0x7fd4101e4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4100b80f0 0x7fd4101e3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd4100b7370 con 0x7fd4100b80f0 2024-03-19T01:24:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd4100b71f0 con 0x7fd4100b1460 2024-03-19T01:24:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd4100b74f0 con 0x7fd41006d870 2024-03-19T01:24:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4100b80f0 0x7fd4101e3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4100b80f0 0x7fd4101e3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33886/0 (socket says 172.21.15.7:33886) 2024-03-19T01:24:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40ffff640 1 -- 172.21.15.7:0/4044803499 learned_addr learned my addr 172.21.15.7:0/4044803499 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40f7fe640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 0x7fd41013ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40effd640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 0x7fd4101e4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40ffff640 1 -- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 msgr2=0x7fd41013ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40ffff640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 0x7fd41013ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40ffff640 1 -- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 msgr2=0x7fd4101e4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40ffff640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 0x7fd4101e4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40ffff640 1 -- 172.21.15.7:0/4044803499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41013d7a0 con 0x7fd4100b80f0 2024-03-19T01:24:37.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40f7fe640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 0x7fd41013ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:37.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40effd640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 0x7fd4101e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:37.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd40ffff640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4100b80f0 0x7fd4101e3fa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd40800c6f0 tx=0x7fd40800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd3f7fff640 1 -- 172.21.15.7:0/4044803499 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd408017070 con 0x7fd4100b80f0 2024-03-19T01:24:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd3f7fff640 1 -- 172.21.15.7:0/4044803499 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd408005db0 con 0x7fd4100b80f0 2024-03-19T01:24:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41013f760 con 0x7fd4100b80f0 2024-03-19T01:24:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd3f7fff640 1 -- 172.21.15.7:0/4044803499 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40800a840 con 0x7fd4100b80f0 2024-03-19T01:24:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.074+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41013d430 con 0x7fd4100b80f0 2024-03-19T01:24:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.078+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd410075460 con 0x7fd4100b80f0 2024-03-19T01:24:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.078+0000 7fd3f7fff640 1 -- 172.21.15.7:0/4044803499 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd40801a070 con 0x7fd4100b80f0 2024-03-19T01:24:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.078+0000 7fd3f7fff640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd3fc067660 0x7fd3fc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.078+0000 7fd3f7fff640 1 -- 172.21.15.7:0/4044803499 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fd408084910 con 0x7fd4100b80f0 2024-03-19T01:24:37.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.078+0000 7fd40f7fe640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd3fc067660 0x7fd3fc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.078+0000 7fd40f7fe640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd3fc067660 0x7fd3fc069b20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd4000b1000 tx=0x7fd400008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:37.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.082+0000 7fd3f7fff640 1 -- 172.21.15.7:0/4044803499 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd4080552d0 con 0x7fd4100b80f0 2024-03-19T01:24:37.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.238+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fd4101e45a0 con 0x7fd3fc067660 2024-03-19T01:24:37.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.238+0000 7fd3f7fff640 1 -- 172.21.15.7:0/4044803499 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd4101e45a0 con 0x7fd3fc067660 2024-03-19T01:24:37.241 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 7 to deep-scrub 2024-03-19T01:24:37.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd3fc067660 msgr2=0x7fd3fc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd3fc067660 0x7fd3fc069b20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd4000b1000 tx=0x7fd400008040 comp rx=0 tx=0).stop 2024-03-19T01:24:37.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4100b80f0 msgr2=0x7fd4101e3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4100b80f0 0x7fd4101e3fa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd40800c6f0 tx=0x7fd40800cbb0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 shutdown_connections 2024-03-19T01:24:37.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7fd3fc067660 0x7fd3fc069b20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4100b80f0 0x7fd4101e3fa0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4100b1460 0x7fd4101e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 --2- 172.21.15.7:0/4044803499 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd41006d870 0x7fd41013ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 >> 172.21.15.7:0/4044803499 conn(0x7fd4100b5330 msgr2=0x7fd4100b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:37.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 shutdown_connections 2024-03-19T01:24:37.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.242+0000 7fd415cf3640 1 -- 172.21.15.7:0/4044803499 wait complete. 2024-03-19T01:24:37.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:24:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.366+0000 7f92b74e5640 1 Processor -- start 2024-03-19T01:24:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.366+0000 7f92b74e5640 1 -- start start 2024-03-19T01:24:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b74e5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b00b81c0 0x7f92b01486e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b74e5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f92b006f1c0 0x7f92b013f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b74e5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92b00b85a0 con 0x7f92b006f640 2024-03-19T01:24:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b74e5640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f92b00b2710 con 0x7f92b006f1c0 2024-03-19T01:24:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b74e5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f92b0070960 con 0x7f92b00b81c0 2024-03-19T01:24:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b5281640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b00b81c0 0x7f92b01486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b4a80640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f92b006f1c0 0x7f92b013f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b5281640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b00b81c0 0x7f92b01486e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.7:48452/0 (socket says 172.21.15.7:48452) 2024-03-19T01:24:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b5281640 1 -- 172.21.15.7:0/2900905325 learned_addr learned my addr 172.21.15.7:0/2900905325 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:37.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b4a80640 1 -- 172.21.15.7:0/2900905325 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b00b81c0 msgr2=0x7f92b01486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:37.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b4a80640 1 --2- 172.21.15.7:0/2900905325 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b00b81c0 0x7f92b01486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b4a80640 1 -- 172.21.15.7:0/2900905325 >> v1:172.21.15.167:6789/0 conn(0x7f92b006f640 legacy=0x7f92b0145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b4a80640 1 -- 172.21.15.7:0/2900905325 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b00b20c0 con 0x7f92b006f1c0 2024-03-19T01:24:37.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92b4a80640 1 --2- 172.21.15.7:0/2900905325 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b006f1c0 0x7f92b013f0a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f92a00097f0 tx=0x7f92a001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a11aa781356e20bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.370+0000 7f92a7fff640 1 -- 172.21.15.7:0/2900905325 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a00ab600 con 0x7f92b006f1c0 2024-03-19T01:24:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92a7fff640 1 -- 172.21.15.7:0/2900905325 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a00b0070 con 0x7f92b006f1c0 2024-03-19T01:24:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92a7fff640 1 -- 172.21.15.7:0/2900905325 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a00ab9c0 con 0x7f92b006f1c0 2024-03-19T01:24:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- 172.21.15.7:0/2900905325 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b006f1c0 msgr2=0x7f92b013f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 --2- 172.21.15.7:0/2900905325 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b006f1c0 0x7f92b013f0a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f92a00097f0 tx=0x7f92a001c3e0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- 172.21.15.7:0/2900905325 shutdown_connections 2024-03-19T01:24:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 --2- 172.21.15.7:0/2900905325 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b006f1c0 0x7f92b013f0a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 --2- 172.21.15.7:0/2900905325 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b00b81c0 0x7f92b01486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- 172.21.15.7:0/2900905325 >> 172.21.15.7:0/2900905325 conn(0x7f92b00b5330 msgr2=0x7f92b00b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:37.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- 172.21.15.7:0/2900905325 shutdown_connections 2024-03-19T01:24:37.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- 172.21.15.7:0/2900905325 wait complete. 2024-03-19T01:24:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 Processor -- start 2024-03-19T01:24:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- start start 2024-03-19T01:24:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b006f1c0 0x7f92b0147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b006f640 0x7f92b01e21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b00b81c0 0x7f92b01dea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f92b0152a90 con 0x7f92b006f640 2024-03-19T01:24:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f92b0152c10 con 0x7f92b00b81c0 2024-03-19T01:24:37.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f92b0152d90 con 0x7f92b006f1c0 2024-03-19T01:24:37.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b4a80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b006f640 0x7f92b01e21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b4a80640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b006f640 0x7f92b01e21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33894/0 (socket says 172.21.15.7:33894) 2024-03-19T01:24:37.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b4a80640 1 -- 172.21.15.7:0/1994940243 learned_addr learned my addr 172.21.15.7:0/1994940243 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:37.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b5a82640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b00b81c0 0x7f92b01dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b5281640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b006f1c0 0x7f92b0147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b4a80640 1 -- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b006f1c0 msgr2=0x7f92b0147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b4a80640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b006f1c0 0x7f92b0147c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b4a80640 1 -- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b00b81c0 msgr2=0x7f92b01dea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b4a80640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b00b81c0 0x7f92b01dea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b4a80640 1 -- 172.21.15.7:0/1994940243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b0142b10 con 0x7f92b006f640 2024-03-19T01:24:37.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b5281640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b006f1c0 0x7f92b0147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:37.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b5a82640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b00b81c0 0x7f92b01dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:37.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b4a80640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b006f640 0x7f92b01e21e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f92a00ad040 tx=0x7f92a001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:37.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92a5ffb640 1 -- 172.21.15.7:0/1994940243 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0006e20 con 0x7f92b006f640 2024-03-19T01:24:37.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92a5ffb640 1 -- 172.21.15.7:0/1994940243 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a00bf610 con 0x7f92b006f640 2024-03-19T01:24:37.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92a5ffb640 1 -- 172.21.15.7:0/1994940243 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a00bf9c0 con 0x7f92b006f640 2024-03-19T01:24:37.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b01e19e0 con 0x7f92b006f640 2024-03-19T01:24:37.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.374+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b00b4030 con 0x7f92b006f640 2024-03-19T01:24:37.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.378+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92b0075460 con 0x7f92b006f640 2024-03-19T01:24:37.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.378+0000 7f92a5ffb640 1 -- 172.21.15.7:0/1994940243 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f92a00bd020 con 0x7f92b006f640 2024-03-19T01:24:37.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.378+0000 7f92a5ffb640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9294067660 0x7f9294069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:37.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.378+0000 7f92a5ffb640 1 -- 172.21.15.7:0/1994940243 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f92a011faf0 con 0x7f92b006f640 2024-03-19T01:24:37.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.378+0000 7f92b5281640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9294067660 0x7f9294069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:37.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.378+0000 7f92b5281640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9294067660 0x7f9294069b20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92ac009ba0 tx=0x7f92ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.382+0000 7f92a5ffb640 1 -- 172.21.15.7:0/1994940243 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f92a0016090 con 0x7f92b006f640 2024-03-19T01:24:37.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.538+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92b013aee0 con 0x7f9294067660 2024-03-19T01:24:37.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.538+0000 7f92a5ffb640 1 -- 172.21.15.7:0/1994940243 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13459 (secure 0 0 0) 0x7f92b013aee0 con 0x7f9294067660 2024-03-19T01:24:37.542 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:37.542 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:24:37.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9294067660 msgr2=0x7f9294069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9294067660 0x7f9294069b20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92ac009ba0 tx=0x7f92ac008040 comp rx=0 tx=0).stop 2024-03-19T01:24:37.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b006f640 msgr2=0x7f92b01e21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:37.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b006f640 0x7f92b01e21e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f92a00ad040 tx=0x7f92a001ac20 comp rx=0 tx=0).stop 2024-03-19T01:24:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 shutdown_connections 2024-03-19T01:24:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f9294067660 0x7f9294069b20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92b00b81c0 0x7f92b01dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b006f640 0x7f92b01e21e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 --2- 172.21.15.7:0/1994940243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92b006f1c0 0x7f92b0147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 >> 172.21.15.7:0/1994940243 conn(0x7f92b00b5330 msgr2=0x7f92b013a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 shutdown_connections 2024-03-19T01:24:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:37.542+0000 7f92b74e5640 1 -- 172.21.15.7:0/1994940243 wait complete. 2024-03-19T01:24:37.561 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-03-19T01:24:36.908374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":236948,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737772,"statfs":{"total":773094113280,"available":772851478528,"internally_reserved":0,"allocated":4235264,"data_stored":2351392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.910825"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-19T01:24:29.168807+0000","last_change":"2024-03-19T01:23:17.284718+0000","last_active":"2024-03-19T01:24:29.168807+0000","last_peered":"2024-03-19T01:24:29.168807+0000","last_clean":"2024-03-19T01:24:29.168807+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:24:29.168807+0000","last_undegraded":"2024-03-19T01:24:29.168807+0000","last_fullsized":"2024-03-19T01:24:29.168807+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:23:15.401918+0000","last_clean_scrub_stamp":"2024-03-19T01:23:15.401918+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:02.126614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31552,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340288,"statfs":{"total":96636764160,"available":96604454912,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:24:37.564 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-19T01:23:15.401918+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=23, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=24, tm_sec=32, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T01:24:37.564 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-19T01:24:53.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:24:53.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:24:57.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:24:57.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.694+0000 7ff80071b640 1 Processor -- start 2024-03-19T01:24:57.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.694+0000 7ff80071b640 1 -- start start 2024-03-19T01:24:57.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff80071b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7f80b81c0 0x7ff7f81486e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:57.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff80071b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f806f1c0 0x7ff7f813f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:57.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff80071b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff7f806f640 0x7ff7f8145170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:57.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff80071b640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 -- 0x7ff7f80b2710 con 0x7ff7f80b81c0 2024-03-19T01:24:57.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff80071b640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7ff7f8070960 con 0x7ff7f806f640 2024-03-19T01:24:57.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff80071b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f80b85a0 con 0x7ff7f806f1c0 2024-03-19T01:24:57.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fe4b7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7f80b81c0 0x7ff7f81486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:57.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fe4b7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7f80b81c0 0x7ff7f81486e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39930/0 (socket says 172.21.15.7:39930) 2024-03-19T01:24:57.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fe4b7640 1 -- 172.21.15.7:0/499396800 learned_addr learned my addr 172.21.15.7:0/499396800 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:57.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fecb8640 1 --2- 172.21.15.7:0/499396800 >> v2:172.21.15.167:3300/0 conn(0x7ff7f806f640 0x7ff7f8145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:57.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fe4b7640 1 -- 172.21.15.7:0/499396800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f806f1c0 msgr2=0x7ff7f813f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:57.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fdcb6640 1 --2- 172.21.15.7:0/499396800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f806f1c0 0x7ff7f813f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:57.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fe4b7640 1 --2- 172.21.15.7:0/499396800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f806f1c0 0x7ff7f813f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fe4b7640 1 -- 172.21.15.7:0/499396800 >> v2:172.21.15.167:3300/0 conn(0x7ff7f806f640 msgr2=0x7ff7f8145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:57.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fe4b7640 1 --2- 172.21.15.7:0/499396800 >> v2:172.21.15.167:3300/0 conn(0x7ff7f806f640 0x7ff7f8145170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fe4b7640 1 -- 172.21.15.7:0/499396800 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f80b20c0 con 0x7ff7f80b81c0 2024-03-19T01:24:57.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fecb8640 1 --2- 172.21.15.7:0/499396800 >> v2:172.21.15.167:3300/0 conn(0x7ff7f806f640 0x7ff7f8145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:57.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fdcb6640 1 --2- 172.21.15.7:0/499396800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f806f1c0 0x7ff7f813f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:57.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.698+0000 7ff7fe4b7640 1 --2- 172.21.15.7:0/499396800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80b81c0 0x7ff7f81486e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009870 tx=0x7ff7f401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b579c96e474a8cc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:57.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fd4b5640 1 -- 172.21.15.7:0/499396800 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40ab640 con 0x7ff7f80b81c0 2024-03-19T01:24:57.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fd4b5640 1 -- 172.21.15.7:0/499396800 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f40b0070 con 0x7ff7f80b81c0 2024-03-19T01:24:57.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fd4b5640 1 -- 172.21.15.7:0/499396800 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40aba00 con 0x7ff7f80b81c0 2024-03-19T01:24:57.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 -- 172.21.15.7:0/499396800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80b81c0 msgr2=0x7ff7f81486e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:57.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 --2- 172.21.15.7:0/499396800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80b81c0 0x7ff7f81486e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009870 tx=0x7ff7f401c460 comp rx=0 tx=0).stop 2024-03-19T01:24:57.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 -- 172.21.15.7:0/499396800 shutdown_connections 2024-03-19T01:24:57.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 --2- 172.21.15.7:0/499396800 >> v2:172.21.15.167:3300/0 conn(0x7ff7f806f640 0x7ff7f8145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 --2- 172.21.15.7:0/499396800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f806f1c0 0x7ff7f813f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 --2- 172.21.15.7:0/499396800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f80b81c0 0x7ff7f81486e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 -- 172.21.15.7:0/499396800 >> 172.21.15.7:0/499396800 conn(0x7ff7f80b5330 msgr2=0x7ff7f80b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:57.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 -- 172.21.15.7:0/499396800 shutdown_connections 2024-03-19T01:24:57.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 -- 172.21.15.7:0/499396800 wait complete. 2024-03-19T01:24:57.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 Processor -- start 2024-03-19T01:24:57.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 -- start start 2024-03-19T01:24:57.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f806f1c0 0x7ff7f8072030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:57.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7f806f640 0x7ff7f81ddc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:57.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f80b81c0 0x7ff7f813e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:57.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f8151c00 con 0x7ff7f806f1c0 2024-03-19T01:24:57.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f8151a80 con 0x7ff7f806f640 2024-03-19T01:24:57.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff80071b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f8151d80 con 0x7ff7f80b81c0 2024-03-19T01:24:57.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fe4b7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f806f1c0 0x7ff7f8072030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:57.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fe4b7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f806f1c0 0x7ff7f8072030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39944/0 (socket says 172.21.15.7:39944) 2024-03-19T01:24:57.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fe4b7640 1 -- 172.21.15.7:0/979162190 learned_addr learned my addr 172.21.15.7:0/979162190 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:57.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fdcb6640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7f806f640 0x7ff7f81ddc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:57.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fecb8640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f80b81c0 0x7ff7f813e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:57.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fe4b7640 1 -- 172.21.15.7:0/979162190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f80b81c0 msgr2=0x7ff7f813e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:57.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fe4b7640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f80b81c0 0x7ff7f813e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fe4b7640 1 -- 172.21.15.7:0/979162190 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7f806f640 msgr2=0x7ff7f81ddc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:57.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fe4b7640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7f806f640 0x7ff7f81ddc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fe4b7640 1 -- 172.21.15.7:0/979162190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f8140cd0 con 0x7ff7f806f1c0 2024-03-19T01:24:57.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.702+0000 7ff7fe4b7640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f806f1c0 0x7ff7f8072030 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40ad040 tx=0x7ff7f401aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:57.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff7eeffd640 1 -- 172.21.15.7:0/979162190 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40be5f0 con 0x7ff7f806f1c0 2024-03-19T01:24:57.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff7eeffd640 1 -- 172.21.15.7:0/979162190 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f40be7b0 con 0x7ff7f806f1c0 2024-03-19T01:24:57.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff7eeffd640 1 -- 172.21.15.7:0/979162190 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40beba0 con 0x7ff7f806f1c0 2024-03-19T01:24:57.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f8140fb0 con 0x7ff7f806f1c0 2024-03-19T01:24:57.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f81e1840 con 0x7ff7f806f1c0 2024-03-19T01:24:57.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f8075460 con 0x7ff7f806f1c0 2024-03-19T01:24:57.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff7eeffd640 1 -- 172.21.15.7:0/979162190 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff7f40b43f0 con 0x7ff7f806f1c0 2024-03-19T01:24:57.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff7eeffd640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff7dc067690 0x7ff7dc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:57.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff7eeffd640 1 -- 172.21.15.7:0/979162190 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7ff7f4120100 con 0x7ff7f806f1c0 2024-03-19T01:24:57.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff7fdcb6640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff7dc067690 0x7ff7dc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:57.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.706+0000 7ff7fdcb6640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff7dc067690 0x7ff7dc069b50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8006fd0 tx=0x7ff7e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:57.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.714+0000 7ff7eeffd640 1 -- 172.21.15.7:0/979162190 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff7f4016110 con 0x7ff7f806f1c0 2024-03-19T01:24:57.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.870+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 --> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7f806b8b0 con 0x7ff7dc067690 2024-03-19T01:24:57.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff7eeffd640 1 -- 172.21.15.7:0/979162190 <== mgr.4102 v2:172.21.15.7:6832/1075636442 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13354 (secure 0 0 0) 0x7ff7f806b8b0 con 0x7ff7dc067690 2024-03-19T01:24:57.875 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:57.876 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-19T01:24:57.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff7dc067690 msgr2=0x7ff7dc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:57.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff7dc067690 0x7ff7dc069b50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8006fd0 tx=0x7ff7e8008040 comp rx=0 tx=0).stop 2024-03-19T01:24:57.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f806f1c0 msgr2=0x7ff7f8072030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:57.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f806f1c0 0x7ff7f8072030 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40ad040 tx=0x7ff7f401aca0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 shutdown_connections 2024-03-19T01:24:57.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7ff7dc067690 0x7ff7dc069b50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7f80b81c0 0x7ff7f813e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7f806f640 0x7ff7f81ddc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 --2- 172.21.15.7:0/979162190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7f806f1c0 0x7ff7f8072030 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:57.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 >> 172.21.15.7:0/979162190 conn(0x7ff7f80b5330 msgr2=0x7ff7f80b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:57.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 shutdown_connections 2024-03-19T01:24:57.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:57.874+0000 7ff80071b640 1 -- 172.21.15.7:0/979162190 wait complete. 2024-03-19T01:24:57.894 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-03-19T01:24:56.913483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":236940,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737780,"statfs":{"total":773094113280,"available":772851486720,"internally_reserved":0,"allocated":4235264,"data_stored":2351392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.0","version":"33'119","reported_seq":203,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-19T01:24:38.075200+0000","last_change":"2024-03-19T01:24:38.075278+0000","last_active":"2024-03-19T01:24:38.075200+0000","last_peered":"2024-03-19T01:24:38.075200+0000","last_clean":"2024-03-19T01:24:38.075200+0000","last_became_active":"2024-03-19T01:23:17.282495+0000","last_became_peered":"2024-03-19T01:23:17.282495+0000","last_unstale":"2024-03-19T01:24:38.075200+0000","last_undegraded":"2024-03-19T01:24:38.075200+0000","last_fullsized":"2024-03-19T01:24:38.075200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'119","last_scrub_stamp":"2024-03-19T01:24:38.074998+0000","last_deep_scrub":"33'119","last_deep_scrub_stamp":"2024-03-19T01:24:38.074998+0000","last_clean_scrub_stamp":"2024-03-19T01:24:38.074998+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:59.667142+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31552,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340288,"statfs":{"total":96636764160,"available":96604454912,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:24:57.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-19T01:24:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f12381b4640 1 Processor -- start 2024-03-19T01:24:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f12381b4640 1 -- start start 2024-03-19T01:24:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f12381b4640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f1230074390 0x7f1230145600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f12381b4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 0x7f123013df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f12381b4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1230144ff0 con 0x7f123013e4c0 2024-03-19T01:24:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f12381b4640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f1230144740 con 0x7f1230074390 2024-03-19T01:24:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f12381b4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f12300701f0 con 0x7f1230144bd0 2024-03-19T01:24:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f123574f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 0x7f123013df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f1235f50640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f1230074390 0x7f1230145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f1236751640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f123013e4c0 0x7f1230147700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.7:56996/0 (socket says 172.21.15.7:56996) 2024-03-19T01:24:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f1236751640 1 -- 172.21.15.7:0/1545549382 learned_addr learned my addr 172.21.15.7:0/1545549382 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f1235f50640 1 -- 172.21.15.7:0/1545549382 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 msgr2=0x7f123013df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f1235f50640 1 --2- 172.21.15.7:0/1545549382 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 0x7f123013df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f1235f50640 1 -- 172.21.15.7:0/1545549382 >> v1:172.21.15.167:6789/0 conn(0x7f123013e4c0 legacy=0x7f1230147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.022+0000 7f1235f50640 1 -- 172.21.15.7:0/1545549382 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1230074770 con 0x7f1230074390 2024-03-19T01:24:58.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f1235f50640 1 --2- 172.21.15.7:0/1545549382 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 0x7f1230145600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1220009d90 tx=0x7f122001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5479574a82bfb848 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:58.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f1234f4e640 1 -- 172.21.15.7:0/1545549382 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12200b2070 con 0x7f1230074390 2024-03-19T01:24:58.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f1234f4e640 1 -- 172.21.15.7:0/1545549382 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12200a5750 con 0x7f1230074390 2024-03-19T01:24:58.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f1234f4e640 1 -- 172.21.15.7:0/1545549382 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12200a5b20 con 0x7f1230074390 2024-03-19T01:24:58.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 -- 172.21.15.7:0/1545549382 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 msgr2=0x7f1230145600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 --2- 172.21.15.7:0/1545549382 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 0x7f1230145600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1220009d90 tx=0x7f122001c970 comp rx=0 tx=0).stop 2024-03-19T01:24:58.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 -- 172.21.15.7:0/1545549382 shutdown_connections 2024-03-19T01:24:58.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 --2- 172.21.15.7:0/1545549382 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 0x7f123013df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 --2- 172.21.15.7:0/1545549382 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 0x7f1230145600 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 -- 172.21.15.7:0/1545549382 >> 172.21.15.7:0/1545549382 conn(0x7f123006bbb0 msgr2=0x7f12300b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:58.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 -- 172.21.15.7:0/1545549382 shutdown_connections 2024-03-19T01:24:58.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 -- 172.21.15.7:0/1545549382 wait complete. 2024-03-19T01:24:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 Processor -- start 2024-03-19T01:24:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.026+0000 7f12381b4640 1 -- start start 2024-03-19T01:24:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f12381b4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 0x7f123013ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f12381b4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f123013e4c0 0x7f123013fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f12381b4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 0x7f12301e06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f12381b4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f12300b2f10 con 0x7f123013e4c0 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f12381b4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f12300b2d90 con 0x7f1230074390 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f12381b4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f12300b5560 con 0x7f1230144bd0 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f123574f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f123013e4c0 0x7f123013fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f123574f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f123013e4c0 0x7f123013fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39952/0 (socket says 172.21.15.7:39952) 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f123574f640 1 -- 172.21.15.7:0/3171116787 learned_addr learned my addr 172.21.15.7:0/3171116787 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f1235f50640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 0x7f123013ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f1236751640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 0x7f12301e06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f123574f640 1 -- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 msgr2=0x7f12301e06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f123574f640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 0x7f12301e06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f123574f640 1 -- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 msgr2=0x7f123013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f123574f640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 0x7f123013ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f123574f640 1 -- 172.21.15.7:0/3171116787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12300b9fd0 con 0x7f123013e4c0 2024-03-19T01:24:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f1236751640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 0x7f12301e06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f1235f50640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 0x7f123013ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f123574f640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f123013e4c0 0x7f123013fa30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f122400e930 tx=0x7f122400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f122e7fc640 1 -- 172.21.15.7:0/3171116787 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1224017070 con 0x7f123013e4c0 2024-03-19T01:24:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f122e7fc640 1 -- 172.21.15.7:0/3171116787 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f122400ca00 con 0x7f123013e4c0 2024-03-19T01:24:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12300b1f30 con 0x7f123013e4c0 2024-03-19T01:24:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f122e7fc640 1 -- 172.21.15.7:0/3171116787 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122400cdb0 con 0x7f123013e4c0 2024-03-19T01:24:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12300b8ff0 con 0x7f123013e4c0 2024-03-19T01:24:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1230075570 con 0x7f123013e4c0 2024-03-19T01:24:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f122e7fc640 1 -- 172.21.15.7:0/3171116787 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f122401a070 con 0x7f123013e4c0 2024-03-19T01:24:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f122e7fc640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f121c067660 0x7f121c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f122e7fc640 1 -- 172.21.15.7:0/3171116787 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f1224085640 con 0x7f123013e4c0 2024-03-19T01:24:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f1235f50640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f121c067660 0x7f121c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.030+0000 7f1235f50640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f121c067660 0x7f121c069b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f12200af000 tx=0x7f12200a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.034+0000 7f122e7fc640 1 -- 172.21.15.7:0/3171116787 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1224056690 con 0x7f123013e4c0 2024-03-19T01:24:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.198+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f123013f650 con 0x7f123013e4c0 2024-03-19T01:24:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.206+0000 7f122e7fc640 1 -- 172.21.15.7:0/3171116787 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) ==== 123+0+0 (secure 0 0 0) 0x7f1224055f90 con 0x7f123013e4c0 2024-03-19T01:24:58.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.206+0000 7f122e7fc640 1 -- 172.21.15.7:0/3171116787 <== mon.0 v2:172.21.15.7:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f122405a0e0 con 0x7f123013e4c0 2024-03-19T01:24:58.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f121c067660 msgr2=0x7f121c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f121c067660 0x7f121c069b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f12200af000 tx=0x7f12200a7070 comp rx=0 tx=0).stop 2024-03-19T01:24:58.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f123013e4c0 msgr2=0x7f123013fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f123013e4c0 0x7f123013fa30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f122400e930 tx=0x7f122400edf0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 shutdown_connections 2024-03-19T01:24:58.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f121c067660 0x7f121c069b20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1230144bd0 0x7f12301e06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f123013e4c0 0x7f123013fa30 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 --2- 172.21.15.7:0/3171116787 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1230074390 0x7f123013ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 >> 172.21.15.7:0/3171116787 conn(0x7f123006bbb0 msgr2=0x7f1230071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:58.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 shutdown_connections 2024-03-19T01:24:58.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.210+0000 7f12381b4640 1 -- 172.21.15.7:0/3171116787 wait complete. 2024-03-19T01:24:58.228 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-19T01:24:58.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:24:58.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a2ba7640 1 Processor -- start 2024-03-19T01:24:58.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a2ba7640 1 -- start start 2024-03-19T01:24:58.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a2ba7640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f409c0af050 0x7f409c0af610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a2ba7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c0aec00 con 0x7f409c0ae820 2024-03-19T01:24:58.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a2ba7640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c0afdd0 con 0x7f409c149d00 2024-03-19T01:24:58.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a2ba7640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f409c0ad910 con 0x7f409c0af050 2024-03-19T01:24:58.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a0943640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f409c0ae820 0x7f409c1495f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41262/0 (socket says 172.21.15.7:41262) 2024-03-19T01:24:58.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a0943640 1 -- 172.21.15.7:0/2844108707 learned_addr learned my addr 172.21.15.7:0/2844108707 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-19T01:24:58.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a1144640 1 --2- 172.21.15.7:0/2844108707 >> v2:172.21.15.167:3300/0 conn(0x7f409c0af050 0x7f409c0af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3978630423 0 0) 0x7f409c0aec00 con 0x7f409c0ae820 2024-03-19T01:24:58.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4088003530 con 0x7f409c0ae820 2024-03-19T01:24:58.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3422169614 0 0) 0x7f4088003530 con 0x7f409c0ae820 2024-03-19T01:24:58.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 >> v2:172.21.15.167:3300/0 conn(0x7f409c0af050 msgr2=0x7f409c0af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:58.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 --2- 172.21.15.7:0/2844108707 >> v2:172.21.15.167:3300/0 conn(0x7f409c0af050 0x7f409c0af610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 >> v1:172.21.15.167:6789/0 conn(0x7f409c149d00 legacy=0x7f409c147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c1490d0 con 0x7f409c0ae820 2024-03-19T01:24:58.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3920349766 0 0) 0x7f40840016a0 con 0x7f409c0ae820 2024-03-19T01:24:58.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 >> v1:172.21.15.7:6789/0 conn(0x7f409c0ae820 legacy=0x7f409c1495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f4088003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4088004810 0x7f4088004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40880052f0 0x7f408800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f409c0ad910 con 0x7f4088003a10 2024-03-19T01:24:58.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f408800c1f0 con 0x7f40880052f0 2024-03-19T01:24:58.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f408800c3f0 con 0x7f4088004810 2024-03-19T01:24:58.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.346+0000 7f40a0943640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f4088003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a1144640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40880052f0 0x7f408800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a0943640 1 -- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4088004810 msgr2=0x7f4088004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:58.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f409bfff640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4088004810 0x7f4088004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a0943640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4088004810 0x7f4088004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a0943640 1 -- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40880052f0 msgr2=0x7f408800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:58.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a0943640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40880052f0 0x7f408800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a0943640 1 -- 172.21.15.7:0/2844108707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4088003530 con 0x7f4088003a10 2024-03-19T01:24:58.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a1144640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40880052f0 0x7f408800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:58.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a1144640 1 -- 172.21.15.7:0/2844108707 reap_dead start 2024-03-19T01:24:58.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f409bfff640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4088004810 0x7f4088004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:58.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a0943640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f4088003e30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f408400c0e0 tx=0x7f408401cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff7868327660535 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:58.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084007c50 con 0x7f4088003a10 2024-03-19T01:24:58.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f40840a9020 con 0x7f4088003a10 2024-03-19T01:24:58.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f409b7fe640 1 -- 172.21.15.7:0/2844108707 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084005b20 con 0x7f4088003a10 2024-03-19T01:24:58.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a2ba7640 1 -- 172.21.15.7:0/2844108707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 msgr2=0x7f4088003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a2ba7640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f4088003e30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f408400c0e0 tx=0x7f408401cb20 comp rx=0 tx=0).stop 2024-03-19T01:24:58.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a2ba7640 1 -- 172.21.15.7:0/2844108707 shutdown_connections 2024-03-19T01:24:58.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a2ba7640 1 --2- 172.21.15.7:0/2844108707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f4088003e30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a2ba7640 1 -- 172.21.15.7:0/2844108707 >> 172.21.15.7:0/2844108707 conn(0x7f409c0b5330 msgr2=0x7f409c0b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:58.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a2ba7640 1 -- 172.21.15.7:0/2844108707 shutdown_connections 2024-03-19T01:24:58.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.350+0000 7f40a2ba7640 1 -- 172.21.15.7:0/2844108707 wait complete. 2024-03-19T01:24:58.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 Processor -- start 2024-03-19T01:24:58.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 -- start start 2024-03-19T01:24:58.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f409c1d64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f409c1e0790 0x7f409c1e0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f409c143060 0x7f409c143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f409c14b6c0 con 0x7f4088003a10 2024-03-19T01:24:58.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f409c14b540 con 0x7f409c1e0790 2024-03-19T01:24:58.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f409c14b840 con 0x7f409c143060 2024-03-19T01:24:58.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a0943640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f409c1d64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a0943640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f409c1d64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39968/0 (socket says 172.21.15.7:39968) 2024-03-19T01:24:58.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a0943640 1 -- 172.21.15.7:0/857326397 learned_addr learned my addr 172.21.15.7:0/857326397 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-19T01:24:58.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a0943640 1 -- 172.21.15.7:0/857326397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f409c143060 msgr2=0x7f409c143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a0943640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f409c143060 0x7f409c143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a0943640 1 -- 172.21.15.7:0/857326397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f409c1e0790 msgr2=0x7f409c1e0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a0943640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f409c1e0790 0x7f409c1e0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a0943640 1 -- 172.21.15.7:0/857326397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c143a50 con 0x7f4088003a10 2024-03-19T01:24:58.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a0943640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f409c1d64b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f408401c950 tx=0x7f4088005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:58.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40997fa640 1 -- 172.21.15.7:0/857326397 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40840b4070 con 0x7f4088003a10 2024-03-19T01:24:58.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40997fa640 1 -- 172.21.15.7:0/857326397 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f40840070d0 con 0x7f4088003a10 2024-03-19T01:24:58.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409c1e1150 con 0x7f4088003a10 2024-03-19T01:24:58.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40997fa640 1 -- 172.21.15.7:0/857326397 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40840074e0 con 0x7f4088003a10 2024-03-19T01:24:58.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409c1d4890 con 0x7f4088003a10 2024-03-19T01:24:58.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f409c075460 con 0x7f4088003a10 2024-03-19T01:24:58.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.354+0000 7f40997fa640 1 -- 172.21.15.7:0/857326397 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f40840bf060 con 0x7f4088003a10 2024-03-19T01:24:58.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.358+0000 7f40997fa640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f407c067690 0x7f407c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:58.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.358+0000 7f40997fa640 1 -- 172.21.15.7:0/857326397 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f408411e740 con 0x7f4088003a10 2024-03-19T01:24:58.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.358+0000 7f409bfff640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f407c067690 0x7f407c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:58.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.358+0000 7f409bfff640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f407c067690 0x7f407c069b50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f408c007fd0 tx=0x7f408c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:58.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.358+0000 7f40997fa640 1 -- 172.21.15.7:0/857326397 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4084016830 con 0x7f4088003a10 2024-03-19T01:24:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.554+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f409c1d5a70 con 0x7f4088003a10 2024-03-19T01:24:58.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.554+0000 7f40997fa640 1 -- 172.21.15.7:0/857326397 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+2219 (secure 0 0 0) 0x7f40840ab2d0 con 0x7f4088003a10 2024-03-19T01:24:58.558 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-19T01:24:58.558 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4753,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6833/3744754940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3744754940},{"type":"v1","addr":"172.21.15.167:6833","nonce":3744754940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5319,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1217128035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":1217128035},{"type":"v1","addr":"172.21.15.7:6835","nonce":1217128035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[]} 2024-03-19T01:24:58.558 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 18 2024-03-19T01:24:58.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f407c067690 msgr2=0x7f407c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f407c067690 0x7f407c069b50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f408c007fd0 tx=0x7f408c009040 comp rx=0 tx=0).stop 2024-03-19T01:24:58.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 msgr2=0x7f409c1d64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:58.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f409c1d64b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f408401c950 tx=0x7f4088005840 comp rx=0 tx=0).stop 2024-03-19T01:24:58.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 shutdown_connections 2024-03-19T01:24:58.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:6832/1075636442,v1:172.21.15.7:6833/1075636442] conn(0x7f407c067690 0x7f407c069b50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f409c143060 0x7f409c143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f409c1e0790 0x7f409c1e0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 --2- 172.21.15.7:0/857326397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4088003a10 0x7f409c1d64b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:58.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 >> 172.21.15.7:0/857326397 conn(0x7f409c0b5330 msgr2=0x7f409c0b4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:58.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 shutdown_connections 2024-03-19T01:24:58.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-19T01:24:58.558+0000 7f40a2ba7640 1 -- 172.21.15.7:0/857326397 wait complete. 2024-03-19T01:24:58.576 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-19T01:24:58.577 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-19T01:24:58.577 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-19T01:24:58.577 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:24:58.588 INFO:tasks.ceph.mds.a:Stopped 2024-03-19T01:24:58.588 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-19T01:24:58.588 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:24:58.607 INFO:tasks.ceph.mds.b:Stopped 2024-03-19T01:24:58.607 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-19T01:24:58.607 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-19T01:24:58.607 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:04.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-03-19T01:25:04.709 INFO:tasks.ceph.osd.0:Stopped 2024-03-19T01:25:04.710 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-19T01:25:04.710 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:04.779 INFO:tasks.ceph.osd.1:Stopped 2024-03-19T01:25:04.779 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-19T01:25:04.779 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:04.853 INFO:tasks.ceph.osd.2:Stopped 2024-03-19T01:25:04.854 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-19T01:25:04.854 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:04.944 INFO:tasks.ceph.osd.3:Stopped 2024-03-19T01:25:04.944 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-19T01:25:04.944 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:05.029 INFO:tasks.ceph.osd.4:Stopped 2024-03-19T01:25:05.030 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-19T01:25:05.030 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:05.116 INFO:tasks.ceph.osd.5:Stopped 2024-03-19T01:25:05.116 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-19T01:25:05.116 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:10.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-19T01:25:10.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-03-19T01:25:11.218 INFO:tasks.ceph.osd.6:Stopped 2024-03-19T01:25:11.218 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-19T01:25:11.218 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:11.273 INFO:tasks.ceph.osd.7:Stopped 2024-03-19T01:25:11.274 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-19T01:25:11.274 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-19T01:25:11.274 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:11.304 INFO:tasks.ceph.mgr.y:Stopped 2024-03-19T01:25:11.304 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-19T01:25:11.304 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:11.332 INFO:tasks.ceph.mgr.x:Stopped 2024-03-19T01:25:11.332 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-19T01:25:11.332 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-19T01:25:11.332 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:11.360 INFO:tasks.ceph.mon.a:Stopped 2024-03-19T01:25:11.360 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-19T01:25:11.361 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:11.392 INFO:tasks.ceph.mon.b:Stopped 2024-03-19T01:25:11.392 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-19T01:25:11.392 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:25:11.456 INFO:tasks.ceph.mon.c:Stopped 2024-03-19T01:25:11.456 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-19T01:25:11.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T01:25:11.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:25:11.474 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-19T01:25:11.777 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:25:11.777 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-19T01:25:11.967 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:25:11.967 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-19T01:25:12.266 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:25:12.267 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-19T01:25:12.433 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:25:12.434 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-19T01:25:12.717 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:25:12.717 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-19T01:25:12.895 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:25:12.896 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-19T01:25:13.070 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:25:13.070 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-19T01:25:13.228 INFO:tasks.ceph:Archiving mon data... 2024-03-19T01:25:13.230 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/data/mon.a.tgz 2024-03-19T01:25:13.231 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-19T01:25:13.237 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.o11BE9uMQq 2024-03-19T01:25:13.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.o11BE9uMQq 2024-03-19T01:25:13.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.o11BE9uMQq 2024-03-19T01:25:13.407 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.o11BE9uMQq is 273KB 2024-03-19T01:25:13.473 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.o11BE9uMQq 2024-03-19T01:25:13.479 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/data/mon.b.tgz 2024-03-19T01:25:13.479 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-03-19T01:25:13.485 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.OotpnEbAKI 2024-03-19T01:25:13.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.OotpnEbAKI 2024-03-19T01:25:13.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.OotpnEbAKI 2024-03-19T01:25:13.673 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.OotpnEbAKI is 368KB 2024-03-19T01:25:13.691 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.OotpnEbAKI 2024-03-19T01:25:13.696 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/data/mon.c.tgz 2024-03-19T01:25:13.696 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-03-19T01:25:13.741 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.EyQbywjuQK 2024-03-19T01:25:13.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.EyQbywjuQK 2024-03-19T01:25:13.871 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.EyQbywjuQK 2024-03-19T01:25:13.932 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.EyQbywjuQK is 368KB 2024-03-19T01:25:13.991 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.EyQbywjuQK 2024-03-19T01:25:13.997 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-19T01:25:13.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-19T01:25:13.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-19T01:25:14.060 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-19T01:25:14.101 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-19T01:25:14.102 INFO:tasks.ceph:Archiving crash dumps... 2024-03-19T01:25:14.103 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/remote/smithi007/crash 2024-03-19T01:25:14.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-19T01:25:14.118 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/remote/smithi167/crash 2024-03-19T01:25:14.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-19T01:25:14.157 INFO:tasks.ceph:Shutting down logrotate 2024-03-19T01:25:14.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:25:14.164 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:25:14.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-19T01:25:14.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-19T01:25:14.288 INFO:tasks.ceph:Compressing logs... 2024-03-19T01:25:14.289 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T01:25:14.308 DEBUG:teuthology.orchestra.run.smithi167:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T01:25:14.328 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27295.log 2024-03-19T01:25:14.328 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32751.log 2024-03-19T01:25:14.329 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35921.log 2024-03-19T01:25:14.330 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32751.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.27295.log.gz 2024-03-19T01:25:14.330 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33061.log 2024-03-19T01:25:14.332 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32751.log.gz 2024-03-19T01:25:14.332 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34681.log 2024-03-19T01:25:14.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34238.log 2024-03-19T01:25:14.333 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33061.log.gz 2024-03-19T01:25:14.333 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35921.log.gz 2024-03-19T01:25:14.333 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37515.log 2024-03-19T01:25:14.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34681.log: /var/log/ceph/ceph-client.0.34238.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-19T01:25:14.333 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34681.log.gz 2024-03-19T01:25:14.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37175.log 2024-03-19T01:25:14.333 INFO:teuthology.orchestra.run.smithi007.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.0.34238.log.gz 2024-03-19T01:25:14.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mgr.y.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37515.log.gz 2024-03-19T01:25:14.334 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37123.log 2024-03-19T01:25:14.334 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37305.log 2024-03-19T01:25:14.338 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33477.log 2024-03-19T01:25:14.339 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37175.log.gz 2024-03-19T01:25:14.339 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37305.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37123.log.gz 2024-03-19T01:25:14.339 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34976.log 2024-03-19T01:25:14.339 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37305.log.gz 2024-03-19T01:25:14.339 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35778.log 2024-03-19T01:25:14.339 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34976.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33477.log.gz 2024-03-19T01:25:14.339 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37383.log 2024-03-19T01:25:14.340 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35778.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34976.log.gz 2024-03-19T01:25:14.340 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32688.log 2024-03-19T01:25:14.340 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35778.log.gz 2024-03-19T01:25:14.340 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35830.log 2024-03-19T01:25:14.340 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35138.log 2024-03-19T01:25:14.340 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37383.log.gz 2024-03-19T01:25:14.340 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35830.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32688.log.gz 2024-03-19T01:25:14.340 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-19T01:25:14.340 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35830.log.gz 2024-03-19T01:25:14.341 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33535.log 2024-03-19T01:25:14.341 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.0.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35138.log.gz 2024-03-19T01:25:14.341 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36108.log 2024-03-19T01:25:14.341 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36728.log 2024-03-19T01:25:14.341 INFO:teuthology.orchestra.run.smithi007.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33535.log.gz 2024-03-19T01:25:14.341 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34785.log 2024-03-19T01:25:14.341 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36728.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36108.log.gz 2024-03-19T01:25:14.341 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32502.log 2024-03-19T01:25:14.342 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36728.log.gz 2024-03-19T01:25:14.352 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37539.log 2024-03-19T01:25:14.353 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34785.log: /var/log/ceph/ceph-client.admin.32502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33278.log 2024-03-19T01:25:14.353 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37539.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34785.log.gz 2024-03-19T01:25:14.353 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.32502.log.gz --verbose 2024-03-19T01:25:14.353 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph-client.admin.35242.log 2024-03-19T01:25:14.353 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37539.log.gz 2024-03-19T01:25:14.353 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35268.log 2024-03-19T01:25:14.353 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35242.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33278.log.gz 2024-03-19T01:25:14.354 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33062.log 2024-03-19T01:25:14.354 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35242.log.gz 2024-03-19T01:25:14.354 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36544.log 2024-03-19T01:25:14.354 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35268.log: /var/log/ceph/ceph-client.admin.33062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33262.log 2024-03-19T01:25:14.354 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36544.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35268.log.gz 2024-03-19T01:25:14.354 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32716.log 2024-03-19T01:25:14.354 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33062.log.gz 2024-03-19T01:25:14.354 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36544.log.gz 2024-03-19T01:25:14.355 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33691.log 2024-03-19T01:25:14.355 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2024-03-19T01:25:14.355 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32716.log.gz 2024-03-19T01:25:14.355 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33262.log.gz 2024-03-19T01:25:14.355 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32911.log 2024-03-19T01:25:14.355 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35190.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33691.log.gz 2024-03-19T01:25:14.355 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34551.log 2024-03-19T01:25:14.355 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-03-19T01:25:14.355 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32989.log 2024-03-19T01:25:14.356 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32911.log: /var/log/ceph/ceph-client.admin.34551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33261.log 2024-03-19T01:25:14.356 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32911.log.gz 2024-03-19T01:25:14.356 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32989.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34551.log.gz 2024-03-19T01:25:14.356 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35700.log 2024-03-19T01:25:14.356 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33261.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32989.log.gz 2024-03-19T01:25:14.356 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37149.log 2024-03-19T01:25:14.356 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33261.log.gz 2024-03-19T01:25:14.356 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35700.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-19T01:25:14.357 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37149.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35700.log.gz 2024-03-19T01:25:14.357 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37149.log.gz 2024-03-19T01:25:14.357 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32358.log 2024-03-19T01:25:14.357 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32803.log 2024-03-19T01:25:14.357 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.32358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27411.log 2024-03-19T01:25:14.357 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32358.log.gz 2024-03-19T01:25:14.357 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36851.log 2024-03-19T01:25:14.358 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27411.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32803.log.gz 2024-03-19T01:25:14.358 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33251.log 2024-03-19T01:25:14.358 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27411.log.gz 2024-03-19T01:25:14.358 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35164.log 2024-03-19T01:25:14.358 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36851.log.gz 2024-03-19T01:25:14.358 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34395.log 2024-03-19T01:25:14.358 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35164.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.33251.log.gz --verbose 2024-03-19T01:25:14.358 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph-client.admin.35886.log 2024-03-19T01:25:14.359 INFO:teuthology.orchestra.run.smithi007.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35164.log.gz 2024-03-19T01:25:14.359 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35524.log 2024-03-19T01:25:14.359 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7%/var/log/ceph/ceph-client.admin.35886.log: -- replaced with /var/log/ceph/ceph-client.admin.34395.log.gz 2024-03-19T01:25:14.359 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34837.log 2024-03-19T01:25:14.359 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0%/var/log/ceph/ceph-client.admin.35524.log: -- replaced with /var/log/ceph/ceph-client.admin.35886.log.gz 2024-03-19T01:25:14.359 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35726.log 2024-03-19T01:25:14.359 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35524.log.gz 2024-03-19T01:25:14.359 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34837.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-19T01:25:14.360 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35726.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34837.log.gz 2024-03-19T01:25:14.360 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32660.log 2024-03-19T01:25:14.360 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35726.log.gz 2024-03-19T01:25:14.360 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36292.log 2024-03-19T01:25:14.360 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33483.log/var/log/ceph/ceph-client.admin.32660.log: 2024-03-19T01:25:14.378 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27527.log 2024-03-19T01:25:14.378 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-19T01:25:14.378 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.27527.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-19T01:25:14.379 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.7.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27527.log.gz 2024-03-19T01:25:14.379 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27503.log 2024-03-19T01:25:14.379 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27551.log 2024-03-19T01:25:14.379 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.27503.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-19T01:25:14.379 INFO:teuthology.orchestra.run.smithi167.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27503.log.gz 2024-03-19T01:25:14.379 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.27551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27623.log 2024-03-19T01:25:14.379 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.5.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27551.log.gz 2024-03-19T01:25:14.379 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27671.log 2024-03-19T01:25:14.379 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.27623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27599.log 2024-03-19T01:25:14.380 INFO:teuthology.orchestra.run.smithi167.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27623.log.gz 2024-03-19T01:25:14.380 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.27671.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-19T01:25:14.380 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-19T01:25:14.380 INFO:teuthology.orchestra.run.smithi167.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27671.log.gz 2024-03-19T01:25:14.380 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.27599.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-19T01:25:14.380 INFO:teuthology.orchestra.run.smithi167.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27599.log.gz 2024-03-19T01:25:14.380 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-19T01:25:14.380 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27647.log 2024-03-19T01:25:14.380 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-19T01:25:14.381 INFO:teuthology.orchestra.run.smithi167.stderr: 91.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-19T01:25:14.381 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.27647.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27647.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-19T01:25:14.381 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-19T01:25:14.381 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-19T01:25:14.381 INFO:teuthology.orchestra.run.smithi167.stderr: 92.5%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-19T01:25:14.381 INFO:teuthology.orchestra.run.smithi167.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-19T01:25:14.381 INFO:teuthology.orchestra.run.smithi167.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-19T01:25:14.381 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27575.log 2024-03-19T01:25:14.381 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.27575.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27575.log.gz 2024-03-19T01:25:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.0.36292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36967.log 2024-03-19T01:25:14.392 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32660.log.gz 2024-03-19T01:25:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36383.log 2024-03-19T01:25:14.392 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33483.log.gz 2024-03-19T01:25:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35294.log 2024-03-19T01:25:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36596.log 2024-03-19T01:25:14.393 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36967.log.gz 2024-03-19T01:25:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35294.log: /var/log/ceph/ceph-client.admin.36383.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36383.log.gz 2024-03-19T01:25:14.393 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35294.log.gz 2024-03-19T01:25:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32634.log 2024-03-19T01:25:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34343.log 2024-03-19T01:25:14.393 INFO:teuthology.orchestra.run.smithi007.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.36292.log.gz 2024-03-19T01:25:14.393 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0%/var/log/ceph/ceph-client.admin.32634.log: -- replaced with /var/log/ceph/ceph-client.admin.36596.log.gz 2024-03-19T01:25:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34291.log 2024-03-19T01:25:14.394 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32634.log.gz 2024-03-19T01:25:14.394 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35216.log 2024-03-19T01:25:14.394 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34343.log.gz 2024-03-19T01:25:14.394 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37045.log 2024-03-19T01:25:14.394 INFO:teuthology.orchestra.run.smithi007.stderr:gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.34291.log.gz --verbose 2024-03-19T01:25:14.394 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph-client.admin.35398.log 2024-03-19T01:25:14.394 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35216.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35216.log.gz 2024-03-19T01:25:14.394 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36461.log 2024-03-19T01:25:14.394 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37227.log 2024-03-19T01:25:14.395 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37045.log.gz 2024-03-19T01:25:14.395 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35398.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35398.log.gz 2024-03-19T01:25:14.395 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37253.log 2024-03-19T01:25:14.395 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34811.log 2024-03-19T01:25:14.395 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4%/var/log/ceph/ceph-client.admin.37227.log: -- replaced with /var/log/ceph/ceph-client.admin.36461.log.gz 2024-03-19T01:25:14.395 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37253.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37227.log.gz 2024-03-19T01:25:14.395 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37487.log 2024-03-19T01:25:14.395 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33276.log 2024-03-19T01:25:14.396 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37253.log.gz 2024-03-19T01:25:14.396 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34811.log: /var/log/ceph/ceph-client.admin.37487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35003.log 2024-03-19T01:25:14.396 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34811.log.gz 2024-03-19T01:25:14.396 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37487.log.gz 2024-03-19T01:25:14.396 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32556.log 2024-03-19T01:25:14.396 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33276.log: /var/log/ceph/ceph-client.admin.35003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35648.log 2024-03-19T01:25:14.396 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35003.log.gz/var/log/ceph/ceph-client.admin.32556.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33276.log.gz 2024-03-19T01:25:14.396 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-19T01:25:14.396 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33769.log 2024-03-19T01:25:14.397 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32556.log.gz 2024-03-19T01:25:14.397 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37331.log 2024-03-19T01:25:14.397 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35648.log.gz 2024-03-19T01:25:14.397 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33769.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-19T01:25:14.397 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2%/var/log/ceph/ceph-client.admin.37331.log: -- replaced with /var/log/ceph/ceph-client.admin.33769.log.gz 2024-03-19T01:25:14.397 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32885.log 2024-03-19T01:25:14.397 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37331.log.gz 2024-03-19T01:25:14.397 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35977.log 2024-03-19T01:25:14.397 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.32885.log: /var/log/ceph/ceph.tmp-client.admin.22531.log 2024-03-19T01:25:14.398 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35977.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32885.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34003.log 2024-03-19T01:25:14.398 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-19T01:25:14.398 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35977.log.gz 2024-03-19T01:25:14.398 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.tmp-client.admin.22531.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22531.log.gz 2024-03-19T01:25:14.398 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34733.log 2024-03-19T01:25:14.398 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33665.log 2024-03-19T01:25:14.398 INFO:teuthology.orchestra.run.smithi007.stderr: 87.5%/var/log/ceph/ceph-client.admin.34733.log: -- replaced with /var/log/ceph/ceph-client.admin.34003.log.gz 2024-03-19T01:25:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35029.log 2024-03-19T01:25:14.399 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34733.log.gz 2024-03-19T01:25:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33665.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33665.log.gz 2024-03-19T01:25:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36186.log 2024-03-19T01:25:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33068.log 2024-03-19T01:25:14.399 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2%/var/log/ceph/ceph-client.admin.36186.log: -- replaced with /var/log/ceph/ceph-client.admin.35029.log.gz 2024-03-19T01:25:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32456.log 2024-03-19T01:25:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33068.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36186.log.gz 2024-03-19T01:25:14.399 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33068.log.gz 2024-03-19T01:25:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34707.log 2024-03-19T01:25:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35569.log 2024-03-19T01:25:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34707.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32456.log.gz 2024-03-19T01:25:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35107.log 2024-03-19T01:25:14.400 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33465.log 2024-03-19T01:25:14.400 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34707.log.gz 2024-03-19T01:25:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35569.log: /var/log/ceph/ceph-client.admin.35107.log: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-19T01:25:14.400 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35569.log.gz 2024-03-19T01:25:14.400 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35107.log.gz 2024-03-19T01:25:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36624.log 2024-03-19T01:25:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33465.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-19T01:25:14.401 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33465.log.gz 2024-03-19T01:25:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36780.log 2024-03-19T01:25:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36702.log 2024-03-19T01:25:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.1.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36624.log.gz 2024-03-19T01:25:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36029.log 2024-03-19T01:25:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36702.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36780.log.gz 2024-03-19T01:25:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32777.log 2024-03-19T01:25:14.402 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34055.log 2024-03-19T01:25:14.402 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36702.log.gz 2024-03-19T01:25:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36029.log: /var/log/ceph/ceph-client.admin.32777.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36029.log.gz 2024-03-19T01:25:14.402 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32777.log.gz 2024-03-19T01:25:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36435.log 2024-03-19T01:25:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27514.log 2024-03-19T01:25:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37071.log 2024-03-19T01:25:14.402 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36435.log.gz 2024-03-19T01:25:14.403 INFO:teuthology.orchestra.run.smithi007.stderr: 91.0%/var/log/ceph/ceph-client.admin.27514.log: -- replaced with /var/log/ceph/ceph-client.admin.34055.log.gz 2024-03-19T01:25:14.403 INFO:teuthology.orchestra.run.smithi007.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.27514.log.gz 2024-03-19T01:25:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36650.log 2024-03-19T01:25:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37461.log 2024-03-19T01:25:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36650.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37071.log.gz 2024-03-19T01:25:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35469.log 2024-03-19T01:25:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36993.log 2024-03-19T01:25:14.403 INFO:teuthology.orchestra.run.smithi007.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36650.log.gz/var/log/ceph/ceph-client.admin.37461.log: 2024-03-19T01:25:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35469.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37461.log.gz 2024-03-19T01:25:14.404 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35469.log.gz 2024-03-19T01:25:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35621.log 2024-03-19T01:25:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35674.log 2024-03-19T01:25:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35621.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36993.log.gz 2024-03-19T01:25:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33015.log 2024-03-19T01:25:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34110.log 2024-03-19T01:25:14.404 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35621.log.gz 2024-03-19T01:25:14.405 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33015.log: /var/log/ceph/ceph-client.admin.35674.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33015.log.gz 2024-03-19T01:25:14.405 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35674.log.gz 2024-03-19T01:25:14.405 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35346.log 2024-03-19T01:25:14.405 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33476.log 2024-03-19T01:25:14.405 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8%/var/log/ceph/ceph-client.admin.35346.log: -- replaced with /var/log/ceph/ceph-client.admin.34110.log.gz 2024-03-19T01:25:14.405 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34924.log 2024-03-19T01:25:14.405 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35346.log.gz 2024-03-19T01:25:14.405 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33476.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33476.log.gz 2024-03-19T01:25:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35752.log 2024-03-19T01:25:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37409.log 2024-03-19T01:25:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35752.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34629.log 2024-03-19T01:25:14.406 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34924.log.gz 2024-03-19T01:25:14.406 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35752.log.gz 2024-03-19T01:25:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37409.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37409.log.gz 2024-03-19T01:25:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-19T01:25:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32528.log 2024-03-19T01:25:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34629.log.gz 2024-03-19T01:25:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33475.log 2024-03-19T01:25:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33977.log 2024-03-19T01:25:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32528.log: /var/log/ceph/ceph-client.admin.33475.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32528.log.gz 2024-03-19T01:25:14.407 INFO:teuthology.orchestra.run.smithi007.stderr: 85.2% -- replaced with /var/log/ceph/ceph.log.gz 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33475.log.gz 2024-03-19T01:25:14.407 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-19T01:25:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36134.log 2024-03-19T01:25:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36929.log 2024-03-19T01:25:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33977.log: /var/log/ceph/ceph-client.admin.36134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34499.log 2024-03-19T01:25:14.408 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33977.log.gz 2024-03-19T01:25:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36929.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36134.log.gz 2024-03-19T01:25:14.408 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36929.log.gz 2024-03-19T01:25:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37279.log 2024-03-19T01:25:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33247.log 2024-03-19T01:25:14.408 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34499.log.gz/var/log/ceph/ceph-client.admin.37279.log: 2024-03-19T01:25:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34447.log 2024-03-19T01:25:14.408 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37279.log.gz 2024-03-19T01:25:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33247.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33247.log.gz 2024-03-19T01:25:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34525.log 2024-03-19T01:25:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36487.log 2024-03-19T01:25:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34525.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34447.log.gz 2024-03-19T01:25:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33899.log 2024-03-19T01:25:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34759.log 2024-03-19T01:25:14.409 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34525.log.gz 2024-03-19T01:25:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33899.log: /var/log/ceph/ceph-client.admin.36487.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33899.log.gz 2024-03-19T01:25:14.428 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36487.log.gz 2024-03-19T01:25:14.428 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34317.log 2024-03-19T01:25:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34421.log 2024-03-19T01:25:14.429 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2%/var/log/ceph/ceph-client.admin.34317.log: -- replaced with /var/log/ceph/ceph-client.admin.34759.log.gz 2024-03-19T01:25:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32426.log 2024-03-19T01:25:14.429 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34317.log.gz 2024-03-19T01:25:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34421.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34421.log.gz 2024-03-19T01:25:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33951.log 2024-03-19T01:25:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36570.log 2024-03-19T01:25:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32426.log: /var/log/ceph/ceph-client.admin.33951.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36056.log 2024-03-19T01:25:14.429 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32426.log.gz 2024-03-19T01:25:14.430 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33951.log.gz 2024-03-19T01:25:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36570.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36570.log.gz 2024-03-19T01:25:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34950.log 2024-03-19T01:25:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36409.log 2024-03-19T01:25:14.430 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0%/var/log/ceph/ceph-client.admin.34950.log: -- replaced with /var/log/ceph/ceph-client.admin.36056.log.gz 2024-03-19T01:25:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32859.log 2024-03-19T01:25:14.431 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34950.log.gz 2024-03-19T01:25:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36409.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36409.log.gz 2024-03-19T01:25:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33925.log 2024-03-19T01:25:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32937.log 2024-03-19T01:25:14.431 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32859.log.gz 2024-03-19T01:25:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33265.log 2024-03-19T01:25:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32937.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33925.log.gz 2024-03-19T01:25:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33821.log 2024-03-19T01:25:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34473.log 2024-03-19T01:25:14.432 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32937.log.gz 2024-03-19T01:25:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33265.log: /var/log/ceph/ceph-client.admin.33821.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33265.log.gz 2024-03-19T01:25:14.432 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33821.log.gz 2024-03-19T01:25:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32608.log 2024-03-19T01:25:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34369.log 2024-03-19T01:25:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32608.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34473.log.gz 2024-03-19T01:25:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-19T01:25:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36754.log 2024-03-19T01:25:14.433 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32608.log.gz 2024-03-19T01:25:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34369.log: /var/log/ceph/ceph.audit.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34369.log.gz 2024-03-19T01:25:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36824.log 2024-03-19T01:25:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33795.log 2024-03-19T01:25:14.433 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-19T01:25:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36754.log: /var/log/ceph/ceph-client.admin.36824.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36754.log.gz 2024-03-19T01:25:14.433 INFO:teuthology.orchestra.run.smithi007.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36824.log.gz 2024-03-19T01:25:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-19T01:25:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37019.log 2024-03-19T01:25:14.434 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7%/var/log/ceph/ceph-mds.a.log: -- replaced with /var/log/ceph/ceph-client.admin.33795.log.gz 2024-03-19T01:25:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36903.log 2024-03-19T01:25:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37019.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37019.log.gz 2024-03-19T01:25:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36082.log 2024-03-19T01:25:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35858.log 2024-03-19T01:25:14.434 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36903.log.gz 2024-03-19T01:25:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36082.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36082.log.gz 2024-03-19T01:25:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33457.log 2024-03-19T01:25:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37565.log 2024-03-19T01:25:14.435 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6%/var/log/ceph/ceph-client.admin.33457.log: -- replaced with /var/log/ceph/ceph-client.admin.35858.log.gz 2024-03-19T01:25:14.435 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33457.log.gz 2024-03-19T01:25:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33717.log 2024-03-19T01:25:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33058.log 2024-03-19T01:25:14.435 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8%/var/log/ceph/ceph-client.admin.33717.log: -- replaced with /var/log/ceph/ceph-client.admin.37565.log.gz 2024-03-19T01:25:14.435 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33717.log.gz 2024-03-19T01:25:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36676.log 2024-03-19T01:25:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36003.log 2024-03-19T01:25:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36676.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33058.log.gz 2024-03-19T01:25:14.436 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36676.log.gz 2024-03-19T01:25:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34898.log 2024-03-19T01:25:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36877.log 2024-03-19T01:25:14.436 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36003.log.gz/var/log/ceph/ceph-client.admin.34898.log: 2024-03-19T01:25:14.436 INFO:teuthology.orchestra.run.smithi007.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34898.log.gz 2024-03-19T01:25:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35495.log 2024-03-19T01:25:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33873.log 2024-03-19T01:25:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35495.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36877.log.gz 2024-03-19T01:25:14.437 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35495.log.gz 2024-03-19T01:25:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36353.log 2024-03-19T01:25:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35947.log 2024-03-19T01:25:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36353.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33873.log.gz 2024-03-19T01:25:14.437 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36353.log.gz 2024-03-19T01:25:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33060.log 2024-03-19T01:25:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33069.log 2024-03-19T01:25:14.438 INFO:teuthology.orchestra.run.smithi007.stderr: 89.5%/var/log/ceph/ceph-client.admin.33060.log: -- replaced with /var/log/ceph/ceph-client.admin.35947.log.gz 2024-03-19T01:25:14.438 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33060.log.gz 2024-03-19T01:25:14.438 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36236.log 2024-03-19T01:25:14.438 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35595.log 2024-03-19T01:25:14.438 INFO:teuthology.orchestra.run.smithi007.stderr: 88.6%/var/log/ceph/ceph-client.admin.36236.log: -- replaced with /var/log/ceph/ceph-client.admin.33069.log.gz 2024-03-19T01:25:14.438 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36236.log.gz 2024-03-19T01:25:14.438 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35055.log 2024-03-19T01:25:14.438 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32582.log 2024-03-19T01:25:14.439 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0%/var/log/ceph/ceph-client.admin.35055.log: -- replaced with /var/log/ceph/ceph-client.admin.35595.log.gz 2024-03-19T01:25:14.439 INFO:teuthology.orchestra.run.smithi007.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35055.log.gz 2024-03-19T01:25:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36518.log 2024-03-19T01:25:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35372.log 2024-03-19T01:25:14.439 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32582.log.gz/var/log/ceph/ceph-client.admin.36518.log: 2024-03-19T01:25:14.439 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36518.log.gz 2024-03-19T01:25:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33059.log 2024-03-19T01:25:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37357.log 2024-03-19T01:25:14.440 INFO:teuthology.orchestra.run.smithi007.stderr: 87.7%/var/log/ceph/ceph-client.admin.33059.log: -- replaced with /var/log/ceph/ceph-client.admin.35372.log.gz 2024-03-19T01:25:14.440 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33059.log.gz 2024-03-19T01:25:14.440 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37097.log 2024-03-19T01:25:14.440 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35440.log 2024-03-19T01:25:14.440 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37097.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37357.log.gz 2024-03-19T01:25:14.440 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37097.log.gz 2024-03-19T01:25:14.440 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33064.log 2024-03-19T01:25:14.440 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36160.log 2024-03-19T01:25:14.440 INFO:teuthology.orchestra.run.smithi007.stderr: 89.5%/var/log/ceph/ceph-client.admin.33064.log: -- replaced with /var/log/ceph/ceph-client.admin.35440.log.gz 2024-03-19T01:25:14.441 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33064.log.gz 2024-03-19T01:25:14.441 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37435.log 2024-03-19T01:25:14.441 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33463.log 2024-03-19T01:25:14.441 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36160.log.gz 2024-03-19T01:25:14.441 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37435.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37435.log.gz 2024-03-19T01:25:14.441 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34577.log 2024-03-19T01:25:14.442 INFO:teuthology.orchestra.run.smithi167.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-19T01:25:14.447 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37201.log 2024-03-19T01:25:14.448 INFO:teuthology.orchestra.run.smithi007.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33463.log.gz/var/log/ceph/ceph-client.admin.34577.log: 2024-03-19T01:25:14.448 INFO:teuthology.orchestra.run.smithi007.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34577.log.gz 2024-03-19T01:25:14.448 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32833.log 2024-03-19T01:25:14.448 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35081.log 2024-03-19T01:25:14.448 INFO:teuthology.orchestra.run.smithi007.stderr: 87.6%/var/log/ceph/ceph-client.admin.32833.log: -- replaced with /var/log/ceph/ceph-client.admin.37201.log.gz 2024-03-19T01:25:14.448 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32833.log.gz 2024-03-19T01:25:14.448 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35804.log 2024-03-19T01:25:14.448 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33847.log 2024-03-19T01:25:14.449 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1%/var/log/ceph/ceph-client.admin.35804.log: -- replaced with /var/log/ceph/ceph-client.admin.35081.log.gz 2024-03-19T01:25:14.449 INFO:teuthology.orchestra.run.smithi007.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35804.log.gz 2024-03-19T01:25:14.449 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34655.log 2024-03-19T01:25:14.449 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32963.log 2024-03-19T01:25:14.449 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34655.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33847.log.gz 2024-03-19T01:25:14.449 INFO:teuthology.orchestra.run.smithi007.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34655.log.gz 2024-03-19T01:25:14.449 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34029.log 2024-03-19T01:25:14.449 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33743.log 2024-03-19T01:25:14.450 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34029.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32963.log.gz 2024-03-19T01:25:14.450 INFO:teuthology.orchestra.run.smithi007.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34029.log.gz 2024-03-19T01:25:14.450 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34603.log 2024-03-19T01:25:14.450 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33269.log 2024-03-19T01:25:14.450 INFO:teuthology.orchestra.run.smithi007.stderr: 89.0%/var/log/ceph/ceph-client.admin.34603.log: -- replaced with /var/log/ceph/ceph-client.admin.33743.log.gz 2024-03-19T01:25:14.450 INFO:teuthology.orchestra.run.smithi007.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34603.log.gz 2024-03-19T01:25:14.450 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35320.log 2024-03-19T01:25:14.450 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34872.log 2024-03-19T01:25:14.451 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5%/var/log/ceph/ceph-client.admin.35320.log: -- replaced with /var/log/ceph/ceph-client.admin.33269.log.gz 2024-03-19T01:25:14.451 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35320.log.gz 2024-03-19T01:25:14.451 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34872.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34872.log.gz 2024-03-19T01:25:14.451 INFO:teuthology.orchestra.run.smithi007.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-19T01:25:14.451 INFO:teuthology.orchestra.run.smithi167.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-19T01:25:14.453 INFO:teuthology.orchestra.run.smithi007.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-19T01:25:14.461 INFO:teuthology.orchestra.run.smithi167.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-19T01:25:14.469 INFO:teuthology.orchestra.run.smithi167.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-19T01:25:14.470 INFO:teuthology.orchestra.run.smithi007.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-19T01:25:14.492 INFO:teuthology.orchestra.run.smithi167.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-19T01:25:14.495 INFO:teuthology.orchestra.run.smithi007.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-19T01:25:14.500 INFO:teuthology.orchestra.run.smithi007.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-19T01:25:14.509 INFO:teuthology.orchestra.run.smithi167.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-19T01:25:14.511 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-19T01:25:14.511 INFO:teuthology.orchestra.run.smithi167.stderr:real 0m0.173s 2024-03-19T01:25:14.511 INFO:teuthology.orchestra.run.smithi167.stderr:user 0m0.607s 2024-03-19T01:25:14.511 INFO:teuthology.orchestra.run.smithi167.stderr:sys 0m0.040s 2024-03-19T01:25:14.539 INFO:teuthology.orchestra.run.smithi007.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-19T01:25:14.541 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-19T01:25:14.541 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m0.230s 2024-03-19T01:25:14.541 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m0.932s 2024-03-19T01:25:14.541 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.100s 2024-03-19T01:25:14.542 INFO:tasks.ceph:Archiving logs... 2024-03-19T01:25:14.542 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/remote/smithi007/log 2024-03-19T01:25:14.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T01:25:14.882 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/remote/smithi167/log 2024-03-19T01:25:14.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T01:25:14.967 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-19T01:25:14.989 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-19T01:25:14.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T01:25:14.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T01:25:15.062 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-19T01:25:15.071 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-19T01:25:15.072 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T01:25:15.074 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T01:25:15.085 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T01:25:15.085 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2024-03-19T01:25:15.085 INFO:teuthology.orchestra.run.smithi007.stdout:*hv01.front.sepi 44.4.53.2 2 u 9 64 77 0.098 -1.864 1.428 2024-03-19T01:25:15.085 INFO:teuthology.orchestra.run.smithi007.stdout:+hv02.front.sepi 63.231.80.2 3 u 14 64 77 0.101 +0.789 1.394 2024-03-19T01:25:15.085 INFO:teuthology.orchestra.run.smithi007.stdout:+hv03.front.sepi 74.6.168.72 3 u 11 64 77 0.223 -0.512 1.335 2024-03-19T01:25:15.086 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:25:15.087 INFO:teuthology.orchestra.run.smithi167.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T01:25:15.087 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================== 2024-03-19T01:25:15.087 INFO:teuthology.orchestra.run.smithi167.stdout:*hv01.front.sepi 44.4.53.2 2 u 18 64 77 0.122 -1.802 0.707 2024-03-19T01:25:15.088 INFO:teuthology.orchestra.run.smithi167.stdout:+hv02.front.sepi 63.231.80.2 3 u 17 64 77 0.100 +0.835 0.693 2024-03-19T01:25:15.088 INFO:teuthology.orchestra.run.smithi167.stdout:+hv03.front.sepi 74.6.168.72 3 u 17 64 77 0.109 -0.409 0.690 2024-03-19T01:25:15.088 INFO:teuthology.orchestra.run.smithi167.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:25:15.089 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-19T01:25:15.098 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-19T01:25:15.098 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-19T01:25:15.107 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-19T01:25:15.115 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-19T01:25:15.124 INFO:teuthology.task.internal:Duration was 775.725290 seconds 2024-03-19T01:25:15.125 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-19T01:25:15.134 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-19T01:25:15.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T01:25:15.137 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T01:25:15.174 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-19T01:25:15.175 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-03-19T01:25:15.175 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T01:25:15.225 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2024-03-19T01:25:15.225 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T01:25:15.235 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-19T01:25:15.236 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T01:25:15.268 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T01:25:15.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-03-19T01:25:15.302 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-19T01:25:15.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T01:25:15.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T01:25:15.400 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-19T01:25:15.410 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-19T01:25:15.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T01:25:15.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T01:25:15.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-19T01:25:15.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:25:15.465 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:25:15.474 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-03-19T01:25:15.487 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2024-03-19T01:25:15.519 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:25:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:25:15.535 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:25:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:25:15.565 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-19T01:25:15.575 INFO:teuthology.task.internal:Transferring archived files... 2024-03-19T01:25:15.576 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/remote/smithi007 2024-03-19T01:25:15.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T01:25:15.599 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297/remote/smithi167 2024-03-19T01:25:15.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T01:25:15.628 INFO:teuthology.task.internal:Removing archive directory... 2024-03-19T01:25:15.628 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T01:25:15.640 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T01:25:15.669 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-19T01:25:15.679 INFO:teuthology.task.internal:Not uploading archives. 2024-03-19T01:25:15.679 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-19T01:25:15.689 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-19T01:25:15.690 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T01:25:15.692 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T01:25:15.696 INFO:teuthology.orchestra.run.smithi007.stdout: 658625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 01:25 /home/ubuntu/cephtest 2024-03-19T01:25:15.717 INFO:teuthology.orchestra.run.smithi167.stdout: 658639 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 01:25 /home/ubuntu/cephtest 2024-03-19T01:25:15.718 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-19T01:25:15.729 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-19T01:25:15.751 INFO:teuthology.nuke:Checking targets against current locks 2024-03-19T01:25:15.777 DEBUG:teuthology.nuke:shortname: smithi007 2024-03-19T01:25:15.777 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T01:25:15.802 DEBUG:teuthology.nuke:shortname: smithi167 2024-03-19T01:25:15.802 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T01:25:15.805 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:03:39.824271', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEk0CATx6EFmnq8G/eetGLUfRIFzePjKl1vLx1XTY7jtRWcro9dJkpYQ7AbkSZ/cqh/1TFSTXT4N26YAozbh5S4='} 2024-03-19T01:25:15.827 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609297', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:03:39.822541', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBAQ+hAwzR5bVVCCOh/YFbz5HEM/C7nGcxQonnlhWFYGHrOLA8v6OS5DUb26YGAKPfdWzQEgzrs0ki802j8QUmc='} 2024-03-19T01:25:15.852 INFO:teuthology.orchestra.console.smithi007:Power off 2024-03-19T01:25:15.853 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T01:25:15.878 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-03-19T01:25:15.878 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:25:15.895 INFO:teuthology.orchestra.console.smithi167:Power off 2024-03-19T01:25:15.895 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T01:25:15.905 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-19T01:25:15.919 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2024-03-19T01:25:15.919 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:25:15.942 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-03-19T01:25:19.907 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:25:19.944 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:25:20.018 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-19T01:25:20.056 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-03-19T01:25:24.022 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:25:24.057 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:25:24.134 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-19T01:25:24.168 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-03-19T01:25:24.269 INFO:teuthology.orchestra.console.smithi167:Power off completed 2024-03-19T01:25:28.138 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:25:28.248 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-03-19T01:25:28.349 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-03-19T01:25:28.454 INFO:teuthology.run:Summary data: description: fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} duration: 775.7252898216248 failure_reason: 'Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c0f678e149b7408f8988b1edd9e9aad3 status: fail success: false 2024-03-19T01:25:28.455 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T01:25:28.542 INFO:teuthology.run:FAIL